Systems and methods for identifying light emitting droplets using an image sensor and lens system

ABSTRACT

An apparatus for identifying a plurality of light emitting droplets includes a detector system having an image sensor comprising an array of pixels and a structure a structure associated with a surface of the image sensor that extends a height away from the surface of the image sensor and defines a field of view for pixels within the array of pixels. The apparatus can additionally include a lens system positioned in front of the image sensor that defines an object space and an image space for the image sensor such that light from a light emitting droplet located in the object space is recorded by a plurality of pixels within the array of pixels.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to, and is a continuation of, PCTApplication No. PCT/EP2019/055592, filed on Mar. 6, 2019, and entitled“SYSTEMS AND METHODS FOR IDENTIFYING LIGHT EMITTING DROPLETS USING ANIMAGE SENSOR AND LENS SYSTEM”, which claims the benefit of and priorityto U.S. Provisional Application No. 62/639,929, filed on Mar. 7, 2018,and entitled “LENSLESS MICROSCOPY APPARATUSES, SYSTEMS AND METHODS”, andU.S. Provisional Application No. 62/805,247, filed on Feb. 13, 2019, andentitled “SYSTEMS AND METHODS FOR IDENTIFYING LIGHT EMITTING DROPLETSUSING AN IMAGE SENSOR AND LENS SYSTEM”, which are incorporated byreference herein in their entirety.

BACKGROUND Technical Field

The present application relates to apparatuses, systems, and methodsincorporating image sensors. More particularly, the present applicationrelates to apparatuses, systems and methods for identifying one or moremicroscopic light emitters within an object space using an image sensorand/or real images thereof within an image space defined by anassociated lens system.

INTRODUCTION

Microscopy is concerned with observing small, often microscopic,objects. Traditional microscopes incorporate a system of lenses tomagnify, and thereby allow viewing of, small objects. In an opticalmicroscope, the system of lenses directs a magnified image of the smallobject to an eyepiece of the microscope while in a digital microscope,the image is focused on an image sensor. The image sensor records orrelays the received image information for display at a monitor. Suchtraditional lens-based microscopes are frequently large and expensive,and owing to the precision required to view microscopic objects,particularly with high resolution, the optics within most lens-basedmicroscopes are delicate. Traditional lens-based microscopes also have alimited depth of field in which objects are focused in an acceptablysharp manner onto, for example, an image sensor. As a consequence,objects that are located at different distances from the lens have to beplaced inside the depth of field of the microscope before they can beinvestigated with the microscope.

More recently, optofluidic microscopes have been investigated that placea specimen directly on an image chip with or without an additional gridof apertures to obtain images from the chip—and do so without a lenssystem. These systems appear to use projection imaging with collimatedillumination light so that they may not be useable imaging, for example,fluorescent objects. Also, the chips used in these applications have alarge pixel pitch, resulting in low resolution. If a grid of aperturesis used with the chips in an effort to increase resolution, itdisadvantageously requires movement of the specimen during imageacquisition.

Also, more recently, artificial apposition compound eyes have beeninvestigated that, imitating insect compound eyes, use an image chipwith a pinhole array on the pixels and a further microlens array thatfocuses light from certain directions on the pinholes. The resolution ofthe artificial compound eyes is quite low because the construction withpinholes and microlenses has a large pitch, and accordingly, a largepitch CMOS sensor is used.

Known microscopic imaging systems are inefficient for concurrent orsequential imaging of light-emitting droplets within a sample vesselwhen the volume of the sample is larger than the space that can bemonitored with the known systems. This is particularly so for measuringand/or monitoring fluorescent droplets within a sample vessel that aregenerated by digital PCR or similar methodologies.

Accordingly, there are a number of problems and disadvantages in thefield of microscopic imaging that can be addressed.

BRIEF SUMMARY

Various embodiments disclosed herein are related to apparatuses,methods, and systems for identifying the number and/or position of aplurality of light emitting droplets located in object space. Suchembodiments beneficially improve microscopy systems, particularlymicroscopy systems for use identifying light emitting droplets generatedby, for example, droplet PCR.

A first aspect provides for an apparatus for identifying a plurality oflight emitting droplets includes a detector system having an imagesensor comprising an array of pixels and a structure a structureassociated with a surface of the image sensor that extends a height awayfrom the surface of the image sensor and defines a field of view forpixels within the array of pixels. The apparatus can additionallyinclude a lens system positioned in front of the image sensor thatdefines an object space and an image space for the image sensor suchthat light from a light emitting droplet located in the object space isin the field of view of a plurality of pixels within the array ofpixels.

In one embodiment, the light emitting droplet is one of a plurality oflight emitting droplets housed within a sample vessel that is disposedwithin the object space. One or more of the plurality of light emittingdroplets can include a nucleic acid and/or a fluorescent markerconfigured to indicate presence of the nucleic acid.

In one embodiment, the plurality of light emitting droplets additionallyinclude PCR reagents. The light emitting droplets can additionallyinclude a droplet identifying fluorescent marker so that the apparatuscan identify each droplet.

In one embodiment, the lens system is configured to project a real imageof the plurality of light emitting droplets in the image space of thelens system with the image space being positioned in front of the imagesensor. The image sensor and the lens system can be configured such thateach of the plurality of light emitting droplets is identifiable andcountable by the detector system.

In one embodiment, the detector system is configured to monitor apresence and/or intensity of fluorescence within the plurality of lightemitting droplets.

In one embodiment, the apparatus can additionally include a thermalcycler for cycling the plurality of light emitting droplets while theplurality of light emitting droplets is located in the object space.

In one embodiment, the apparatus can additionally include at least aportion of a microfluidic device positioned such that the object spaceis disposed within at least a portion of the microfluidic device. Theapparatus can additionally include a light source, such as an LED,configured to illuminate the object space and/or a beam splitter that ispositioned in the path of illumination light provided by the lightsource to the object space and in the path of light emitted by theplurality of light emitting droplets and recorded by the detectorsystem.

In one embodiment, the light source can include two or more individuallight sources, such as two or more individual LEDs, positioned proximatethe detector system and around an optical axis of the lens system thatare configured to illuminate the object space from different directions.The light source can provide light of two or more different wavelengthranges for exciting different fluorescent markers.

In one embodiment, a density of the plurality of light emitting dropletsis greater than or less than a density of the continuous phase so thatthe plurality of light emitting droplets tends to aggregate at apredefined space within the object space.

The detector system of the disclosed apparatuses can be an integratedunit. The apparatus can additionally include a plurality of detectorsystems and a plurality of object spaces such that each object space ismonitored by at least one detector system. In embodiments having aplurality of object spaces, the plurality of object spaces can have apitch of 9 mm or a multiple of 9 mm, and a total number of object spacescan be 96 or a divisor of 96.

In one embodiment, the image space defined between the image sensor andthe lens system can include a real image of the sample vessel. The realimage of the sample vessel can be located in the field of view of theplurality of pixels such that the image sensor records light from thereal image of the sample vessel. The lens system can include one or morestandard lenses, one or more biconvex lenses, one or more Fresnellenses, or a combination thereof to form a convergent lens system or atelecentric lens system (e.g., a telecentric lens system having amagnification of the image space of about 1).

In embodiments where the apparatus includes a telecentric lens system,the telecentric lens system may include two or more convergent lenssystems with each adjacent pair of convergent lens systems being spacedapart a distance equal to the sum of the focal lengths of eachconvergent lens system. Each adjacent pair of convergent lens systemscan include a proximate convergent lens system and a distal convergentlens system such that the proximate convergent lens system is positionedbetween the image sensor and the distal convergent lens system and thefocal length of the proximate convergent lens system is less than orequal to the focal length of the distal convergent lens system.

In one embodiment, an axial magnification of the adjacent pair ofconvergent lens systems increases the object space in a directionperpendicular to the image sensor. For example, the object space can beextended between 0.5 mm and 20 mm in a direction perpendicular to theimage sensor. Additionally, or alternatively, the image space can be atleast two times smaller (e.g., between two and fifty times smaller,between six and ten times smaller, or more than fifty times smaller)than the object space in a direction perpendicular to the image sensor.

As provided above, the apparatus can include an image sensor having astructure associated therewith. The structure may be associated with asurface of the image sensor and include a plurality of walls positionedon pixel boundaries, forming a regular grid (e.g., around each pixel oraround a set of pixels, comprising groups of neighboring pixels). In oneembodiment, the plurality of walls additionally includes a horizontalstructure disposed on top of the walls such that a cross-section of awall within the plurality of walls forms a T-profile.

The image sensor can have a pixel pitch between about 0.8 μm-10 μm, suchas about 1 μm, and the height of the structure associated with the imagesensor can be between about 0.4 μm-30 μm or between 0.5 and 3 times thepixel pitch of the image sensor. The image sensor can additionallyinclude comprises a backside illuminated chip for increasing a lightsensitive area of each pixel or each set of pixels within the array ofpixels.

In one embodiment, the apparatus further includes one or more opticalfilters associated with the structure to limit light recorded at thearray of pixels to one or more wavelength ranges. The one or moreoptical filters can be arranged in a color filter array, for example, sothat adjacent pixels record light of different wavelength ranges. In oneaspect, the structure includes or is made at least partially of alow-reflective or non-reflective material, such as a metal and/or othermaterial with a high absorption coefficient.

Embodiments of the present disclosure additionally include systems formonitoring fluorescence in a sample vessel. Such systems can include aplurality of apparatuses, as summarized above and described in greaterdetail below in addition to a multi-well plate. The sample vesselassociated with each apparatus of the plurality of apparatuses caninclude a well within the multi-well plate.

In one embodiment, the system is a system for monitoring dropletpolymerase chain reaction (PCR). Such a system may include a samplevessel having a plurality of droplets such that each droplet of theplurality of droplets includes PCR reagents and a subset of theplurality of droplets includes nucleic acid. The system can additionallyinclude a thermal cycler operably connected with the sample vessel and adetector system made up of at least an image sensor, a structureassociated with the image sensor and defining a field of view for theimage sensor, and a lens system positioned within the field of view anddefining an image space for the image sensor. The detector system can bepositioned such that the sample vessel is within the image space and areal image of the plurality of droplets is imagable by the image sensor.

In one embodiment, the system for monitoring droplet PCR includes animage processor configured to identify and/or count a number oflight-emitting droplets within the plurality of droplets. Counting thelight-emitting droplets can include capturing a light intensity value ateach photoactivated pixel of the image sensor and identifying localmaxima of light intensity based on the light intensity values.Alternatively, counting the number of light-emitting droplets caninclude capturing a light intensity value at each photoactivated pixelof the image sensor, identifying a first light-emitting droplet having alocal maximum of light intensity and a corresponding light profileassociated with the local maximum, subtracting the local maximum and thecorresponding light profile from the captured light intensity values,and identifying subsequent light-emitting droplets for each remaininglocal maximum and corresponding light profile. Additionally, oralternatively, the image processor can be configured to identify arelative concentration of nucleic acid between each light-emittingdroplet within the subset of droplets.

In one embodiment, the system includes a color filter array fordetecting at least two different wavelengths of light.

In one embodiment, each droplet of the plurality of droplets includes atleast a first and second fluorophore. The first fluorophore can beconfigured to identify each of the plurality of droplets and the secondfluorophore can be configured to identify the subset of dropletscomprising nucleic acid. Alternatively, the first fluorophore canidentify a first multiplex PCR amplification product and the secondfluorophore can identify a second multiplex PCR amplification product.

Embodiments of the present disclosure additionally include methods foridentifying light-emitting droplets. In one embodiment, a method foridentifying light-emitting droplets includes at least the step ofthermal cycling a sample vessel having a plurality of droplets. Thedroplets can include PCR reagents and a nucleic-acid-bindingfluorophore, and at least a subset of the plurality of droplets caninclude nucleic acid. The method can additionally include monitoring thesample vessel using a detector system that includes an image sensor, astructure associated with the image sensor and defining a field of viewfor the image sensor, and a lens system positioned within the field ofview and defining an image space for the image sensor such that thesample vessel is positioned within the image space of the detectorsystem. The method can additionally include capturing a light intensityvalue at each photoactivated pixel of the image sensor receiving lightfrom a real image of the sample vessel disposed in the image space andidentifying a number of light-emitting droplets based on the lightintensity values.

In one embodiment, the step of identifying a number of light-emittingdroplets includes counting a number of local maxima of light intensity.Alternatively, identifying a number of light-emitting droplets caninclude (i) determining a local maximum of light intensity and acorresponding light profile associated with the local maximum, (ii)subtracting the local maximum and the corresponding light profile fromthe captured light intensity values, and (iii) repeating steps (i) and(ii) for each subsequent local maximum and corresponding light profile.

Embodiments of the present disclosure additionally include methods foridentifying a light emitting droplet disposed in object space. In oneembodiment, a method for identifying a light emitting droplet disposedin object space includes (i) providing an image sensor including anarray of pixels arranged in rows and columns and a structure associatedwith and extending a height away from a surface of the image sensor, thestructure defining a field of view for each pixel within the array ofpixels, (ii) determining a light intensity value for each of a pluralityof photoactivated pixels, the plurality of photoactivated pixelsreceiving light from the light emitting droplet disposed in the objectspace, and (iii) identifying a first photoactivated pixel having a localmaximum of light intensity, the first photoactivated pixel being closerto the light emitting droplet than other pixels of the plurality ofphotoactivated pixels receiving less light than the first photoactivatedpixel.

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the detaileddescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an indication of the scope of the claimed subject matter.

Additional features and advantages of the disclosure will be set forthin the description which follows, and in part will be obvious from thedescription, or may be learned by the practice of the disclosure. Thefeatures and advantages of the disclosure may be realized and obtainedby means of the instruments and combinations particularly pointed out inthe appended claims. These and other features of the present disclosurewill become more fully apparent from the following description andappended claims or may be learned by the practice of the disclosure asset forth hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above recited and otheradvantages and features of the disclosure can be obtained, a moreparticular description of the disclosure briefly described above will berendered by reference to specific embodiments thereof, which areillustrated in the appended drawings. It is appreciated that thesedrawings depict only typical embodiments of the disclosure and are nottherefore to be considered to be limiting of its scope. The disclosurewill be described and explained with additional specificity and detailthrough the use of the accompanying drawings in which:

FIG. 1 illustrates an example embodiment of a system incorporatingfeatures disclosed or envisioned herein;

FIG. 2 illustrates a schematic of an image sensor having a structureassociated therewith such that the structure surrounds each pixel,forming a regular grid;

FIG. 3 is a cross-sectional side view of the image sensor and associatedstructure of FIG. 2, showing a light emitter disposed a height above theimage sensor;

FIG. 4 illustrates an exemplary field of view for a pixel within thearray of pixels comprising the image sensor, as limited by theassociated structure;

FIG. 5 is a diagram illustrating a side view of two light emitters andcorresponding cones of light recorded by pixels of the image sensor andthe resultant exemplary combined light intensity profile of the twolight emitters;

FIG. 6 is another diagram illustrating a side view of two light emittersand corresponding cones of light recorded by pixels of the image sensorand the resultant exemplary combined light intensity profile of the twolight emitters;

FIG. 7 is yet another diagram illustrating a side view of two lightemitters and corresponding cones of light recorded by pixels of theimage sensor and the resultant exemplary combined light intensityprofile of the two light emitters;

FIG. 8 is still another diagram illustrating a side view of two lightemitters and corresponding cones of light recorded by pixels of theimage sensor and the resultant exemplary combined light intensityprofile of the two light emitters;

FIG. 9 illustrates a schematic of an image sensor having a structureassociated therewith such that the structure surrounds 3-by-3 groups ofpixels, forming a regular grid;

FIG. 10 is a cross-sectional side view of the image sensor andassociated structure of FIG. 9, showing a light emitter disposed aheight above the image sensor;

FIG. 11 illustrates a schematic of the image sensor of FIGS. 2-4associated with an exemplary lens system; and

FIG. 12 illustrates a schematic of the image sensor of FIGS. 2-4associated with another exemplary lens system;

FIG. 13 illustrates a general schematic of a thermal cycling and imagingsystem for identifying a plurality of light emitting droplets;

FIG. 14 illustrates an exemplary embodiment of a sample vessel having aplurality of light emitting droplets identified by a detector system;

FIG. 15 illustrates an exemplary embodiment of a sample vessel having aplurality of light-emitting droplets where a real image of thelight-emitting droplets is identified by the associated image sensor ofthe displayed detector system;

FIG. 16 is a diagram illustrating the plurality of light-emittingdroplets from FIG. 14 or 15 and corresponding cones of light recorded bypixels of the associated image sensor and the resultant exemplarycombined light intensity profile of the plurality of light-emittingdroplets; and

FIG. 17 illustrates an exemplary graph of fluorescence intensityexhibited by a plurality of light-emitting droplets during a thermalcycling process, as measured by an imaging system associated therewith.

DETAILED DESCRIPTION

As used in the specification, a word appearing in the singularencompasses its plural counterpart, and a word appearing in the pluralencompasses its singular counterpart, unless implicitly or explicitlyunderstood or stated otherwise. Furthermore, it is understood that forany given component or embodiment described herein, any of the possiblecandidates or alternatives listed for that component may generally beused individually or in combination with one another, unless implicitlyor explicitly understood or stated otherwise. Additionally, it will beunderstood that any list of such candidates or alternatives is merelyillustrative, not limiting, unless implicitly or explicitly understoodor stated otherwise. In addition, unless otherwise indicated, numbersexpressing quantities, constituents, distances, or other measurementsused in the specification and claims are to be understood as beingmodified by the term “about.”

Accordingly, unless indicated to the contrary, the numerical parametersset forth in the specification and attached claims are approximationsthat may vary depending upon the desired properties sought to beobtained by the subject matter presented herein. At the very least, andnot as an attempt to limit the application of the doctrine ofequivalents to the scope of the claims, each numerical parameter shouldat least be construed in light of the number of reported significantdigits and by applying ordinary rounding techniques. Notwithstandingthat the numerical ranges and parameters setting forth the broad scopeof the subject matter presented herein are approximations, the numericalvalues set forth in the specific examples are reported as precisely aspossible. Any numerical values, however, inherently contain certainerrors necessarily resulting from the standard deviation found in theirrespective testing measurements.

Furthermore, as used in the specification and appended claims,directional terms, such as “top,” “bottom,” “left,” “right,” “up,”“down,” “upper,” “lower,” “proximal,” “adjacent,” “distal,” and the likeare used herein solely to indicate relative directions and are nototherwise intended to limit the scope of the specification or claims.

Overview of Imaging Systems and Methods

Embodiments disclosed or envisioned herein may comprise or utilize aspecial purpose or general-purpose computer including computer hardware,such as, for example, one or more processors, as discussed in greaterdetail below. Embodiments may also include physical and othercomputer-readable media for carrying or storing computer-executableinstructions and/or data structures. Such computer-readable media can beany available media that can be accessed by a general purpose or specialpurpose computer system. Computer-readable media that storecomputer-executable instructions are physical storage media.Computer-readable media that carry computer-executable instructions aretransmission media. Thus, by way of example, and not limitation,embodiments can comprise at least two distinctly different kinds ofcomputer-readable media: computer storage media and transmission media.

Computer storage media includes RAM, ROM, EEPROM, CD-ROM or otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium which can be used to store desired programcode means in the form of computer-executable instructions or datastructures and which can be accessed by a general purpose or specialpurpose computer.

A “network” is defined as one or more data links that enable thetransport of electronic data between computer systems and/or modulesand/or other electronic devices. When information is transferred orprovided over a network or another communications connection (eitherhardwired, wireless, or a combination of hardwired and wireless) to acomputer, the computer properly views the connection as a transmissionmedium. Transmission media can include a network and/or data links whichcan be used to carry data or desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer. Combinationsof the above should also be included within the scope ofcomputer-readable media.

Further, upon reaching various computer system components, program codemeans in the form of computer-executable instructions or data structurescan be transferred automatically from transmission media to computerstorage media (or vice versa). For example, computer-executableinstructions or data structures received over a network or data link canbe buffered in RAM within a network interface module (e.g., an “NIC”),and then eventually transferred to computer system RAM and/or to lessvolatile computer storage media at a computer system. Thus, it should beunderstood that computer storage media can be included in computersystem components that also (or even primarily) utilize transmissionmedia.

Computer-executable instructions comprise, for example, instructions anddata which cause a general-purpose computer, special purpose computer,or special purpose processing device to perform a certain function orgroup of functions. The computer executable instructions may be, forexample, binaries, intermediate format instructions such as assemblylanguage, or even source code. Although the subject matter has beendescribed in language specific to structural features and/ormethodological acts, it is to be understood that the subject matterdefined in the appended claims is not necessarily limited to thedescribed features or acts described above. Rather, the describedfeatures and acts are disclosed as example forms of implementing theclaims.

Those skilled in the art will appreciate that embodiments may bepracticed in network computing environments with many types of computersystem configurations, including, personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, tablets, smart phones,routers, switches, and the like. Embodiments may be practiced indistributed system environments where local and remote computer systems,which are linked (either by hardwired data links, wireless data links,or by a combination of hardwired and wireless data links) through anetwork, both perform tasks. In a distributed system environment,program modules may be located in both local and remote memory storagedevices. Program modules for one entity can be located and/or run inanother entities data center or “in the cloud.” In this specificationand in the following claims, a computer system is also defined toinclude imaging systems (e.g., imaging system 102 in FIG. 1).

FIG. 1 illustrates an exemplary system 100 incorporating featuresdisclosed or envisioned herein. At the heart of the system 100 is animaging system 102 in which samples, such as biological cells, areimaged and analyzed. The exemplary imaging system 102 includes, but isnot limited to, an image sensor assembly 104 and a computing device 106.Within the image sensor assembly 104 is an image sensor (e.g., imagesensor 114 of FIG. 2) configured to capture image data from a samplepositioned within the field of view of the image sensor. In a generalworking example, a sample 110 is placed directly on the image sensorassembly 104 or otherwise positioned within the field of view of theimage sensor within assembly 104. The image sensor captures image datafrom the sample 110, which is further analyzed, as described below, tocreate a three-dimensional (3D) image of the object space containing thesample and components disposed therein. In a preferred embodiment, theimage sensor assembly 104 captures image data from one or more lightemitters within the sample, and based on the image data, constructs a 3Dimage of the observed object space, including the positions of lightemitters within the object space with sub-pixel resolution. In thefollowing, the term light emitter may be used interchangeably with theterm light emitting droplet.

According to embodiments, light emitters or light emitting droplets mayemit light because they include molecules or substances that emit light.Light emission may be based on luminescence such as, for example,chemiluminescence or fluorescence. Light emitters may include, forexample, fluorophores or quantum dots. Light emitters may include targetmolecules (e.g., DNA, RNA, proteins, inorganic molecules) or targetobjects (e.g., cells, beads, larger structures or certain parts oflarger structures) and they may include light emitting molecules orsubstances associated with the target molecules or target objects toidentify the target molecules or target objects and/or to count them orto reconstruct the position of the light emitters (i.e. light emittingdroplets) in three-dimensional space.

Embodiments of the present disclosure can identify light emitters usinga single image of a volume of space that is large in comparison with avolume that, for example, a traditional light microscope can sharplyfocus onto an image sensor. In an example, different types of lightemitters may be used to identify different targets. The light emittersmay emit light at different characteristic wavelengths, and embodimentsof the present disclosure may be able to identify the different lightemitters by their characteristic wavelengths. This may be achieved, forexample, in embodiments using a color filter array above the pixels ofthe image sensor. As an additional example, embodiments of the presentdisclosure may also take multiple images in given time period (e.g., 24or 30 images per second) to record any movement of the light emittersand/or any change in light emission intensity.

As shown in FIG. 1, a stage housing 108 can be mounted on or otherwisebe associated with the image sensor assembly 104 to facilitateillumination and/or positioning of the sample 110. The sample can beincluded within or mounted on any sample receiving apparatus, including,for example, a microscope slide 110 a, a multi-well plate (e.g., a96-well plate 110 b shown in FIG. 1), a flow cell 110 c, or similar.Accordingly, the stage housing 108 can include one or more light sourcesto illuminate the sample 110, which can be, for example, a white lightor a light of a defined wavelength. In embodiments where the lightemitter includes a fluorophore, the light source can include afluorophore excitation light source. For example, the stage housing 108can include a light engine comprising multiple light emitting diodes(LEDs) configured to emit an excitation wavelength for excitingfluorophores within the sample 110. Additionally, or alternatively, thestage housing 108 can include optical filters that filter the excitationand emission light, such as a multi-position dichroic filter wheeland/or a multi-position emission filter wheel.

As a general method of operation, a fluorophore excitation source can beautomatically or manually directed to provide multiple bandwidths oflight ranging from violet (e.g., 380 nm) to near infrared (e.g., atleast 700 nm) and are designed to excite fluorophores, such as, forexample, cyan fluorescent protein (CFP) and Far Red (i.e., near-IR)fluorophores. Example LED bandwidths with appropriate excitation filters(e.g., as selected via a computing device 106 driven excitation filterwheel) can include, but are not limited to, Violet (380-410 nm LED &386/23 nm excitation filter), Blue (420-455 nm LED & 438/24 nmexcitation filter), Cyan (460-490 nm LED & 485/20 nm excitation filter),Green (535-600 nm LED & 549/15 nm excitation filter), Green (535-600 nmLED & 560/25 nm excitation filter), Red (620-750 nm LED & 650/13 nmexcitation filter), and Near-IR (700-IR nm LED & 740/13 nm excitationfilter). The two Green/excitation filter combinations listed above canbe provided optionally via, for example, a mechanical flipper, whendesiring to improve the brightness of red and scarlet dyes. Of course,other LED bandwidths can also be used.

Additionally, or alternatively, the stage housing 108 can include astage assembly and positioning mechanism configured to retain andselectively move sample for viewing by the image sensor, as known in theart. As it should be appreciated, the stage assembly can be configuredto move within any of three-dimensions, as known in the art. Forexample, the stage assembly can be configured to move laterally (e.g.,in an x, y-plane parallel to the surface of the image sensor) toposition different portions of the sample within the field of view ofthe image sensor. The stage assembly can additionally, or alternatively,be configured to move in a z-direction (e.g., between parallel x,y-planes that are each disposed at different distances from the surfaceof the image sensor) using any mechanism known in the art, such as, forexample, a stepper motor and screw/nut combination providing step-wisemovements of the sample toward/away from the image sensor in incrementsdown to 0.006 μm/microstep.

In some embodiments, it can be advantageous to control or adjust thedistance between the sample, or the closest light emitter within thesample, and the image sensor of the assembly 104—or in other words, toadjust the object space viewed by the image sensor within assembly 104.Doing so, it may be possible to adjust the object space such that adesired number of light emitters (or other portion of the sample) is inthe field of view of a desired number or percentage of pixels within thearray of pixels defining the image sensor. For example, the stageassembly can position the sample such that a desired number of lightemitters (or defined portion of the sample) is in the field of view ofat least nine pixels and/or of less than 90% of the pixels within thearray of pixels defining the image sensor. Doing so can optimize and/orincrease the accuracy of the system in determining the three-dimensionalposition of light emitters within the object space, as detailed morefully below.

Upon capturing image data at the image sensor, the data can be analyzedand/or stored locally at the image sensor assembly 104 and/or inconjunction with the computing device 106. This can include, forexample, constructing the position of light emitters within thesample/object space. The computing device 106 can additionally be usedas a controller for the system as well as for performing, by itself orin conjunction with image sensor assembly 104, the analysis and/orstorage of data obtained by image sensor assembly 104. Computing device106 can comprise a general purpose or specialized computer or server orthe like, as defined above, or any other computerized device. Computingdevice 106 can communicate with image sensor assembly 104 directly orthrough a network, as is known in the art. In some embodiments,computing device 106 is incorporated into the image sensor assembly 104.In some embodiments, the computing device is incorporated within theimage sensor assembly.

System 100 can also include a user display device 112 to display resultsand/or system configurations. Image sensor assembly 104 and/or computingdevice 106 can communicate, either directly or indirectly, with userdisplay device 112 and can cause the position of the light emitterwithin object space to be displayed on the user display device 112. Forexample, the computing device 106 can construct a 3D image of theobserved object space with each identified light emitter positionedwithin the 3D image.

In one embodiment, one or more of the method steps described herein areperformed as a software application. However, embodiments are notlimited to this and method steps can also be performed in firmware,hardware or a combination of firmware, hardware and/or software.Furthermore, the steps of the methods can exist wholly or in part on theimage sensor assembly 104, computing device 106, and/or other computingdevices.

An operating environment for the devices of the system may comprise orutilize a processing system having one or more microprocessors andsystem memory. In accordance with the practices of persons skilled inthe art of computer programming, embodiments are described below withreference to acts and symbolic representations of operations orinstructions that are performed by the processing system, unlessindicated otherwise. Such acts and operations or instructions arereferred to as being “computer-executed,” “CPU-executed,” or“processor-executed.”

Exemplary Image Sensor Assemblies

As described above, the imaging system 102 can include an image sensorassembly 104. As shown in FIG. 2, the image sensor assembly 104 caninclude an image sensor 114 and a structure 116 associated with thesurface of the image sensor 114. The image sensor 114 can be any CCD orCMOS sensor array or chip having an array of pixels arranged in rows andcolumns, as shown, for example, in the magnified view (A) of FIG. 2.

In the exemplary embodiment of FIG. 2, the structure 116 includes aplurality of walls 118 positioned on the pixel boundaries, forming aregular grid. Accordingly, each pixel 120 of the array of pixels isbounded by walls 118 of the structure 116. It should be appreciated,however, that while the structure 116 of FIG. 2 illustrates walls 118positioned on each pixel boundary such that each individual pixel 120 isbounded by walls 118, other arrangements or spacing of the walls can beused. For example, the walls of the structure could bound a set ofpixels (e.g., a 2×1, 2×2, 2×3, 3×3, or larger set of pixels).Additionally, or alternatively, the walls could be positioned off of thepixel boundary, within the light receiving area of the pixels. In apreferred embodiment, the walls of the structure are positioned on thepixel boundaries.

To overcome the limitations of prior art image sensors, which fail tomaintain high resolution capacity while obtaining image data ofstationary objects for determining the position of the objects in objectspace, the structures disclosed herein extend a height away from thesurface of the image sensor and thereby define a field of view for eachpixel. This enables an object, such as a light emitter, to bepositionally located without requiring the object to move relative tothe image sensor. For example, based on the number and location ofpixels receiving light from a light emitter for a given image sensor andstructure, the z-distance between the surface of the image sensor andthe light emitter can be determined along with the x- and y-coordinateswith respect to the image sensor. Using these data, thethree-dimensional object space comprising the light emitter can bemodeled, along with the position of the light emitter within the objectspace.

Applications of this technology can be used to improve variousmicroscopy systems and methods. For example, a 3D model is traditionallyobtained by compiling z-sequences of optical slices of a sample (e.g.,using confocal laser scanning microscopy or traditional wide-fieldmicroscopy), but this requires imaging each of a plurality of focalplanes and compiling or stacking these image sections to form the 3Dimage stack comprising the model. Accordingly, the position of lightemitters, such as fluorescing portions of the sample, within the 3Dmodel are given a contextual position within the sample by comparingimages of adjacent focal planes. Instead of capturing a host of imagesat different focal planes and assembling these images to render a 3Dmodel, the systems of the present disclosure can—with a singleimage—identify the three-dimensional position of objects within theobject space containing at least a portion of the corresponding sample.

Clearly, the systems and methods disclosed herein offer significantadvantages over prior art microscopy systems and methods. Because asingle image is sufficient to render the three-dimensional positions ofobjects within a sample, there is no need to move the sample through aseries of z-sequences, and therefore, expensive motors required forpositioning the sample at each optical slice of the z-sequence are notnecessary. This reduces the cost and mechanical and operationalcomplexity of microscopy systems. Images can be obtained more quicklyand with a reduced digital storage cost (e.g., because an associatedcomputing system only stores data corresponding to a single imageinstead of data corresponding to stacks of images and the spatialrelationships between each image of the stack).

Additional advantages are seen particularly within applications offluorescence microscopy. Photobleaching is a well-known problem influorescence microscopy; in essence, the longer a fluorophore is exposedto excitation light, the less light it emits until it no longer emitslight in response to excitation. This is particularly problematic ininstances where a large sample is imaged or where multiple focal planesare imaged. For large samples, fluorophores outside of the viewing areaoften receive incident excitation radiation, causing photobleaching ofresponsive fluorophores outside the viewing area, and when thesefluorophores are eventually imaged, their fluorescence intensity isreduced from their original state. This can limit the usefulness of thedata. For example, such photobleaching can make it difficult to quantifyand compare fluorescence intensity of objects between viewing areas of alarge image.

Similarly, in situations where a series of images are captured atdifferent focal planes, excitation light is directed at a single viewingarea for a prolonged period of time, and images captured later withinthe z-sequence are likely to suffer from photobleaching—again limitingthe usefulness of the data. Additionally, the resolution of opticalslices or the resultant 3D model can be limited by photobleaching. Thatis, resolution can be dependent upon a combination of how quickly thefluorophores photobleach and how quickly each optical slice can becaptured. A faster capture rate of optical slices often results in agreater number of low-resolution optical slices. This allows for moreprecise positioning of objects within the sample but comes at the costof a lower resolution for each optical slice. On the other hand, asmaller number of high-resolution slices offers greater clarity at eachoptical slice but comes at the cost of a lower resolution 3D model ofthe sample and less precise positioning of objects within the sample.

The disclosed systems and methods beneficially reduce the amount of timefluorophores are exposed to excitation light without sacrificing—or insome cases increasing—the precision by which the position of objects,such as light emitters, can be determined within the sample. Further,because the location of fluorescent objects within the sample can bedetermined quickly and without significant photobleaching, the systemsand methods disclosed herein can enable the image sensor to image adesired optical volume quickly and precisely. For example, a singleimage can be captured and the position of a light emitter determinedtherefrom independently of where the light emitter is located within thedefined object space. In a further example, the whole object space canbe monitored by reading out the image sensor multiple times in a giventime period (e.g., 24 or 30 times a second). This can enable thedisclosed system to record movement of one or more light emitters withinthe object space and/or changes in the intensity of light emitted by theone or more light emitters within the object space.

With continued reference to FIG. 2, an image sensor 114 having regularrectangular walls 118 around each pixel 120 is shown. In other words,the walls 118 are positioned on the pixel boundaries and form a regulargrid. A cross-sectional side view (taken along line (B)) of the imagesensor 114 and associated structure 116 is shown in FIG. 3. Referringthereto, it is possible to calculate the area of each pixel 120 thatreceives light from a light emitter 122 depending on the position of thelight emitter 122 with respect to the array of pixels 120 of the imagesensor 114. In an approximation, the product of the area of a pixel 120receiving light and the solid angle of the light receiving area of thepixel 120 with respect to the light emitter 122 is assumed to beproportional to the light intensity measured by the pixel 120. Using themeasured values of the pixels and the calculation, it is possible todetermine the position of the light emitters 122 in three-dimensionalobject space and with further input also the brightness of the lightemitters 122. In any one of the embodiments, a light emitter may be anobject or substance that is capable of providing light. A light emittermay be, e.g. a luminescent substance such as a fluorescent substance orfluorescent molecule or it may be illuminated object. The light providedby a light emitter may include light of the visible wavelength range,the infrared wavelength range, and/or the ultraviolet wavelength range.The light may include the wavelength range that is measurable by imagesensors with pixels.

An exemplary calculation is described below with reference to the lightemitter 122 and image sensor assembly 104 of FIG. 3.

As shown in the cross-sectional side view of image sensor assembly 104,p is the pixel pitch, l is the side length of the light sensitive areaof the pixels 120 (p=1+d with d being the thickness of the walls 118), his the height of the walls 118, p₀ is the pixel 120 which is closest tothe light emitter 122 (i.e., the pixel 120 is located at the positionthrough which a line 124 that is perpendicular to the image sensorpasses between the image sensor 114 and the light emitter 122), thepixels p₁ are the pixels 120 that are i pixels far from p₀ in a row (orcolumn) of the image sensor 114. q₀ is the distance in the −i directionbetween the center of pixel p₀ and the line 124 that is perpendicular tothe image sensor 114 and goes through the light emitter 122 (q₀ can bepositive or negative). s₁ is the length of the shadow that is cast bythe wall 118 between pixel p₀ and p₁. α_max is the maximal angle atwhich light from the light emitter 122 is received at the pixels 120. α₁is the angle between the wall 118 between pixel p₀ and p₁ and the lightthat just reaches the light sensitive area of the pixel p₁, and it isalso the angle between the line 124 and the light that just reaches thelight sensitive area of the pixel p₁.

In some embodiments, the walls 118 are made of or include alow-reflective material, preferably a non-reflective material. Thematerial may additionally, or alternatively, have a high absorptioncoefficient or can be metal which additionally, or alternatively, canabsorb the light. Correction factors taking into account, for example,reflections may be used in the following calculations if this is not thecase.

One factor of the light intensity that is measured by a pixel p_(i) isproportional to the area of the light sensitive area of the pixel thatreceives light from the light emitter 122. As a simplification, it canbe assumed that the light sensitive area includes all of the bottom ofthe wells (i.e., the area between the walls) and that the measurementefficiency of the light sensitive area does not depend on the angle atwhich the light hits on the image sensor 114 given the restricted rangeof angles at which light is measured. If these assumptions are notjustified, further correction factors may be included in the followingcalculation.

The distance of the light emitter 122 to the bottom of the wells istaken as r so that the distance of the light emitter 122 to the top ofthe wells is r−h. The shadow length s_(i) for pixel p_(i),i>0, is thefollowing:

  s_(i) = h ⋅ tan (α_(i))${\tan ( \alpha_{i} )} = {{( {q_{0} + {l\text{/}2} + {i \cdot d} + {( {i - 1} ) \cdot l}} )\text{/}( {r - h} )} = {( {q_{0} - \frac{l}{2} + {i \cdot p}} )\text{/}( {r - h} )}}$

From this follows

s _(i) =h·(q ₀ −l/2+i·p)I(r−h).

For i<0, it is

$\begin{matrix}{s_{i} = {{h \cdot ( {{- q_{0}} - \frac{l}{2} - {i \cdot p}} )}\text{/}( {r - h} )}} \\{{= {{h \cdot ( {{{{{sgn}(i)} \cdot {q\_}}0} - {l\text{/}2} + {{i} \cdot p}} )}\text{/}( {r - h} )}},}\end{matrix}$

where sgn(x) is +1 for x≥0 and −1 for x≤0 and abs(x) is +x for x≥0 and−x for x<0.

For i≠0, it is then

s _(i) =h·(sgn(i)·q ₀ −l/2+|i|·p)/(r−h).

For i≠0, it is

s _(i) =h·(|q ₀ |−l/2)·step(|q ₀ |−l/2)/(r−h),

where step(x) is 0 for x<0 and 1 for x≥0. The reason for this is thatthe light emitter 122 casts only a shadow in the i-direction of theclosest pixel if the light emitter 122 is positioned above the wall 118which is positioned at l/2 from the pixel center.

The area of pixel p_(i), i>0, that measures light from the light emitter122 is proportional to l−s_(i). It follows for the side length α₁ ofpixel p_(i) that receives light from the light emitter 122

$\begin{matrix}{\alpha_{i} = {l - s_{i}}} \\{= {l - {{h \cdot ( {{{{sgn}(i)} \cdot q_{0}} - {l\text{/}2} + {{i} \cdot p}} )}\text{/}( {r - h} )}}} \\{{= {( {1\text{/}( {r - h} )} ) \cdot ( {l - {h \cdot ( {{{{sgn}(i)} \cdot q_{0}} - {l\text{/}2} + {{i} \cdot p}} )}} )}},{{{for}\mspace{14mu} i} \neq 0.}}\end{matrix}$

This result is only valid for i≠0 andtan(α_(i))=(sgn(i)·q₀+l/2+|i|·p)/(r−h)<(l/h)(=tan(α_max)). From thesecond limitation follows the upper absolute bound for |i|:|i<(r·l/h−sgn(i)·q₀−l/2)/p.

For i=0, it is

a _(i) =l−h·(q ₀ −l/2)·step(q ₀ −l/2)/(r−h).

The same calculation can be made for the direction perpendicular to thei-direction of the image sensor 114 (i.e., in the direction of thecolumn or row respectively) to give the following result for the areaa_(i,j) of the pixels that receives light from the light emitter 122that has as the closest pixel p_(i) ₀ _(,j) ₀ :

$a_{i,j} = {( \frac{1}{( {r - h} )^{2}} ) \cdot \lbrack {{r \cdot l} - {h \cdot ( {{{{{sgn}( {i - i_{0}} )} \cdot}q_{0,i}} + \frac{l}{2} + {{{i - i_{0}}} \cdot p}} )}} \rbrack \cdot {\quad{\lbrack {{r \cdot l} - {h \cdot ( {{{{sgn}( {j - j_{0}} )} \cdot q_{0,j}} - \frac{l}{2} + {{{j - j_{0}}} \cdot p}} )}} \rbrack = {\quad{{\lbrack {l - {h \cdot \frac{\begin{pmatrix}\begin{matrix}{{sgn}{( {i - i_{0}} ) \cdot}} \\{q_{0,i} - \frac{l}{2} +}\end{matrix} \\{{{i - i_{0}}} \cdot p}\end{pmatrix}}{r - h}}} \rbrack \cdot \lbrack {l - {h \cdot \frac{\begin{pmatrix}\begin{matrix}{{sgn}{( {j - j_{0}} ) \cdot}} \\{q_{0,j} - \frac{l}{2} +}\end{matrix} \\{{{j - j_{0}}} \cdot p}\end{pmatrix}}{r - h}}} \rbrack},\mspace{20mu} {{{〚*〛}\mspace{14mu} {for}\mspace{20mu} {{i - i_{0}}}} > {0\mspace{14mu} {and}\mspace{14mu} {{j,{- j_{0}}}}} > {0\mspace{14mu} {and}\mspace{14mu} {for}\mspace{20mu} {{i - i_{0}}}} < {\lbrack \frac{( {\frac{rl}{h} - {{{sgn}( {i - i_{0}} )} \cdot q_{0,i}} - \frac{l}{2}} )}{p} \rbrack \mspace{14mu} {and}\mspace{20mu} {{j - j_{0}}}} < {\lbrack \frac{( {\frac{rl}{h} - {{{sgn}( {j - j_{0}} )} \cdot q_{0,j}} - \frac{l}{2}} )}{p} \rbrack.}}}}}}}$

For i=i₀ and |i−i₀|>0, it is

$a_{i_{0},j} = {\lbrack {l - {h \cdot \frac{\begin{matrix}{( {{q_{0,i}} - \frac{l}{2}} ) \cdot} \\{{step}( {{q_{0,i}} - \frac{l}{2}} )}\end{matrix}}{r - h}}} \rbrack \cdot {\quad{\lbrack {l - {h \cdot \frac{\begin{pmatrix}\begin{matrix}{{sgn}{( {j - j_{0}} ) \cdot}} \\{q_{0,j} - \frac{l}{2} +}\end{matrix} \\{{{j - j_{0}}} \cdot p}\end{pmatrix}}{r - h}}} \rbrack.{〚*〛}}}}$

For j=j₀ and |j₀|>0, it is

$a_{i,j_{0}} = {\lbrack {l - {h \cdot \frac{\begin{pmatrix}\begin{matrix}{{sgn}{( {i - i_{0}} ) \cdot}} \\{q_{0,i} - \frac{l}{2} +}\end{matrix} \\{{{{i - i_{0}}} \cdot p}}\end{pmatrix}}{r - h}}} \rbrack \cdot {\quad{\lbrack {l - {h \cdot \frac{\begin{matrix}{( {{q_{0,j}} - \frac{l}{2}} ) \cdot} \\{{step}( {{q_{0,j}} - \frac{l}{2}} )}\end{matrix}}{r - h}}} \rbrack.{〚*〛}}}}$

For i=i₀ and j=j₀, it is

$a_{i_{0,}j_{0}} = {\lbrack {l - {h \cdot \frac{\begin{matrix}{( {{q_{0,i}} - \frac{l}{2}} ) \cdot} \\{{step}( {{q_{0,i}} - \frac{l}{2}} )}\end{matrix}}{r - h}}} \rbrack \cdot {\lbrack {l - {h \cdot \frac{\begin{matrix}{( {{q_{0,j}} - \frac{l}{2}} ) \cdot} \\{{step}( {{q_{0,j}} - \frac{l}{2}} )}\end{matrix}}{r - h}}} \rbrack.{〚*〛}}}$

As a special case, a_(i) ₀ _(,j) ₀ =l² for |q_(0,i)|<l/2 and|q_(0,j)|<l/2.

Equations

*

describe the light sensitive area that records light from a single lightemitter 122 that is point like (has no extension which is a goodapproximation of a light emitter that is much smaller than, for example,the pixel pitch).

It should be appreciated that FIG. 3 can be viewed as a cross-section ofeither a row or a column of pixels 120 of the image sensor 114. As such,in each of the foregoing equations

*

above, q_(0,i) is the distance measured in the descending row directionbetween the point on the image sensor surface closest to the lightemitter and the center of the closest photoactivated pixel and q_(0,j)is a distance measured in the descending column direction between thepoint on the image sensor surface closest to the light emitter and thecenter of the closest photoactivated pixel.

Equations

*

enable designing the image sensor based on the desired observationspace. For this, one can assume that i₀=0, j₀=0, q_(0,j)=0, q_(0,j)=0,d=0 (i.e., l=p, which is a good approximation for thin walls 118), andr−h=r (which is good for distances much greater than wall height) toobtain the approximation

a _(i,j)=(p−h/r)·(−p/2+|i|·p))·(p−(h/r)·(−p/2+|j|·p)).

From this follows directly that the number of pixels receiving light ini- or j-directions is proportional to 2·(r/h) and that the number oftotal pixels receiving light is proportional to 4·(r/h)². These numbersare independent of the pixel pitch, p, and thus, the height of thewalls, h, has to be designed in accordance with the possible distances,r, of the light emitters from the image sensor for an image sensor witha given number of pixels. For example, it may be useful to design theimage sensor so that at least nine pixels record light from the closestlight emitter and no more than, e.g., 90% of the pixels record lightfrom the furthest light emitter. In another example, it may be useful todesign the image sensor so that at least 36 pixels record light from theclosest light emitter and no more than, e.g., 1,000,000 pixels recordlight from the furthest light emitter. This means in some embodimentsthat the values for r are restricted approximately to being greater than2 h and smaller than 1000 h, preferably greater than 3 h and smallerthan 500 h.

In embodiments without a lens system between the object space and theimage sensor, very small values for r (such as 2 h or 3 h) may beachieved by applying a very thin transparent layer (which has athickness of 2 h or 3 h, respectively) above the image sensor. The lightemitters may then be able to get as close to the image sensor as thetransparent layer allows. In embodiments with a lens system (discussedin greater detail below), the confinements of the object space candefine the confinements of the image space and by selecting the objectspace accordingly, the possible distances of the real images of thelight emitters from the image sensor can be limited to, for example, 2 hor 3 h.

In different embodiments, the smallest value for r may be 5 h, 10 h, 100h, or 500 h, corresponding to values of around 5 μm to up to 1,000 μm(depending on h). In embodiments without a lens system, larger valuesfor r may be a consequence of a thicker layer that protects the imagesensor better from the light emitters and the environment in which thelight emitters are located (e.g., a liquid or a transparent substance).However, in embodiments with or without a lens system, the minimal r maybe selected in such a way that at least a certain minimal number ofpixels record light from each light emitter. The minimal number ofpixels may be, for example, nine, 36, 100, 400, or 10,000.

Equations

*

describe the main component of the light profile measured by the imagesensor from a single light emitter. As can be seen in the equations

*

, the measured light intensity has its maximum value at pixel p_(i) ₀_(,j) ₀ and the main component described by equations

*

decreases linearly by going away from the closest pixel p_(i) ₀ _(j) ₀into i and j directions (rows and columns of the image sensor).

The foregoing concept is illustrated in FIG. 4. As shown forrepresentative pixel p₀, a field of view 126 for pixels 120 within theimage sensor 114 is defined and limited by the height of the walls 118and the side length 128 of a light receiving area of the pixels 120.Because some of the object space is precluded from the field of view126, it is possible for a light profile to be generated for each lightemitter that includes a subset of pixels within the array that receiveslight from the light emitter. In embodiments where the structure forms aregular grid, the corresponding light profile will be uniform in shapewith the outer pixels measuring a lower light intensity than pixelslocated at the center of the light profile. The column 130 illustratesthe part of the field of view 126 in which light emitters are recordedwith a maximal intensity, and the closer the light emitter is to thelimit of the field of view, the more the intensity is reduced such thata center or closest pixel can be identified. This demonstrates, at leastin part, how the light profile for a given light emitter has anintensity gradient, and further, how a center or closest pixel can bedetermined and/or calculated.

With equations

*

, it is already possible to determine the position of the light emitter122 because the pixel which is closest to the single light emitter 122(i.e., the pixel, p₀ through which the perpendicular line 124 throughthe plane of the image sensor 114 and the single light emitter 122 goes)indicates the position of the light emitter 122 in two directions andthe distance r of the light emitter 122 from the image sensor 114 can becalculated with equations

*

using measured pixel values. For example, the closest pixel can bedetermined by identifying the pixel receiving the most light or byidentifying the pixel that is in the middle of a pixel pair on the linei=i₀ or j=j₀, where each pixel of the pair records nearly the sameamount of light but with light recording increasing in one direction forone of the two pixels and light recording decreasing in the samedirection for the other one of the two pixels.

The distance r can be determined by identifying the distance between theclosest pixel and the last pixel in either i or j direction thatreceives light from the light emitter. The condition tan(α₁)<tan(α_max)can be used to calculate r depending on the maximal number of pixels inone direction from the closest pixel which still receives light from thelight emitter i_max. With the approximation that Q_(0,i)=0, it followsin the i-direction

r=h·(i_max·(p/l)+½).

It is interesting to note that this result for r is not affected by anattenuation of the light travelling from the light emitter 122 to theimage sensor 114 as long as the furthest pixels that still receivemeasurable light are not changed.

In one embodiment, the light recording profile of a light emitter 122may be determined more accurately, for example as described in moredetail below, and r may be determined by using the measurements ofpixels that are closer to the closest pixel and extrapolate themeasurements to determine i_max (or j_max). In one or more otherembodiments, the light profile of a light emitter may be determinedpurely through measurements, and this light profile may then be used toidentify locations of light emitters in three-dimensional space withreal measurements.

Therefore, the position of the single light emitter 122 can bedetermined in the three-dimensional object space using the equations

*

.

This determination can be used to generate an image of the object spaceand determine the position of the single light emitter 122. As such, theforegoing may be, in some embodiments, sufficient for applications withsingle light emitters or clearly separated light emitters like, e.g.,counting fluorescent objects in flow cytometry.

Theoretically and within an accuracy of around half a pixel pitch, it ispossible to determine the distance r of a single light emitter 122 tothe image sensor 114 by a measurement of a single pixel 120 assumingthat the closest pixel and some other factors are known or have beendetermined. Also, the distance r can also be determined with theaccuracy of around half a pixel pitch from the number of pixels thatrecord light from the single light emitter and this calculation does noteven depend on the brightness of the single light emitter.

However, it is possible to determine the distance r and the positions inthe other two directions with sub-pixel accuracy (given by q_(0,i) andq_(0,j)) by taking into account measurements from more than one pixel.

Generally, the measured light intensity w_(i,j) of pixel i,j can becalculated as follows for a completely clear liquid (i.e., a liquid thatdoes not absorb light):

w _(i,j) =w ₀·Ω_(i,j) ·a _(i,j),

**

with Ω_(i,j) being the solid angle of the light emitter with respect tothe light receiving area of pixel i,j and w₀ being a proportionalityfactor depending, for example, on the light intensity of the lightemitter and on the measurement efficiency of the pixels.

The solid angle Ω_(i,j) can be calculated as known in the art. Forexample, the solid angle Ω_(i,j) of a light emitter with respect to thelight receiving area of a pixel p_(i,j) can be calculated using theformula for the solid angle Ω_(pyr) of a peak or a pyramid located inheight h above the center of a rectangle with side lengths a and b:

${\Omega_{pyr}( {a,b,h} )} = {4{{\arctan\lbrack {a \cdot \frac{b}{( {2{h \cdot \sqrt{( {2h} )^{2} + a^{2} + b^{2}}}} )}} \rbrack}.}}$

The solid angle of the light receiving area of a pixel p_(i,j) that isnot directly centered below the light emitter can now be determinedusing Ω_(pyr) and by calculating the solid angles for larger basis areaswhich form a pyramid with the light emitter centered and subtractingbasis areas which do not belong to the light receiving area of the pixelp_(i,j). This strategy is explained in an article titled “Solid Angle ofa Rectangular Plate” by Richard J. Mathar (dated May 18, 2015, availableonline from the Max-Planck Institute of Astronomy). For i≠i₀ and j≠j₀,it is

$\Omega_{i,j} = {( \frac{1}{4} ) \cdot {\{ {{\Omega_{pyr}\lbrack {{2 \cdot ( {{{{sgn}( {i - i_{0}} )} \cdot q_{0,i}} + \frac{l}{2} + {{{i - i_{0}}} \cdot p}} )},{2 \cdot ( {{{{sgn}( {j - j_{0}} )} \cdot q_{0,j}} + \frac{l}{2} + {{{j - j_{0}}} \cdot p}} )},r} \rbrack} - {\Omega_{pyr}\lbrack {{2 \cdot ( {{{{sgn}( {i - i_{0}} )} \cdot q_{0,i}} + \frac{l}{2} + {{{i - i_{0}}} \cdot p} + s_{i}} )}\ ,{2 \cdot ( {{{{sgn}( {j - j_{0}} )} \cdot q_{0,j}} + \frac{l}{2} + {{{j - j_{0}}} \cdot p}} )},r} \rbrack} - {\Omega_{pyr}\lbrack {{2 \cdot ( {{{{sgn}( {i - i_{0}} )} \cdot q_{0,i}} + \frac{l}{2} + {{{i - i_{0}}} \cdot p}} )}\ ,{2 \cdot ( {{{{sgn}( {j - j_{0}} )} \cdot q_{0,j}} + \frac{l}{2} + {{{j - j_{0}}} \cdot p} + s_{j}} )},r} \rbrack} + {\Omega_{pyr}\lbrack {{2 \cdot ( {{{{sgn}( {i - i_{0}} )} \cdot q_{0,i}} - \frac{l}{2} + {{{i - i_{0}}} \cdot p} + s_{i}} )}\ ,{2 \cdot ( {{{{sgn}( {j - j_{0}} )} \cdot q_{0,j}} - \frac{l}{2} + {{{j - j_{0}}} \cdot p} + s_{j}} )},r} \rbrack}} \}.}}$

s₁ and s_(j) are the shadows calculated above:

$s_{i} = {h \cdot \frac{( {{{{sgn}( {i - i_{0}} )} \cdot q_{0,i}} - \frac{l}{2} + {{{i - i_{0}}} \cdot p}} )}{r - h}}$$s_{j} = {h \cdot {\frac{( {{{{sgn}( {j - j_{0}} )} \cdot q_{0,j}} - \frac{l}{2} + {{{j - j_{0}}} \cdot p}} )}{r - h}.}}$

For i=i₀ and j=j₀ and |q_(0,i)|<l/2 and |q_(0,j)|<l/2, which is a goodapproximation for thin walls, it is

$\Omega_{i_{0},j_{0}} = {( \frac{1}{4} ) \cdot \{ {{\Omega_{pyr}\lbrack {{2 \cdot ( {q_{0,i} + \frac{l}{2}} )},{2 \cdot ( {q_{0,j} + \frac{l}{2}} )},r} \rbrack} + { \quad{{\Omega_{pyr}\lbrack {{2 \cdot ( {{- q_{0,i}} + \frac{l}{2}} )},{2 \cdot ( {q_{0,j} + \frac{l}{2}} )}, r} \rbrack} + {\Omega_{pyr}\lbrack {{2 \cdot ( {q_{0,i} + \frac{l}{2}} )},{2 \cdot ( {{- q_{0,j}} + \frac{l}{2}} )},r} \rbrack} + {\Omega_{pyr}\lbrack {{2 \cdot ( {{- q_{0,i}} + \frac{l}{2}} )},{2 \cdot ( {{- q_{0,j}} + \frac{l}{2}} )},r} \rbrack}} \}.}} }$

For i≠i₀ and j=j₀ and |q_(0,j)|≤l/2, it is

$\Omega_{i,j} = {( \frac{1}{4} ) \cdot {\{ {{\Omega_{pyr}\lbrack {{2 \cdot ( {{{{sgn}( {i - i_{0}} )} \cdot q_{0,i}} + \frac{l}{2} + {{{i - i_{0}}} \cdot p}} )},{2 \cdot ( {q_{0,j} + \frac{l}{2}} )},r} \rbrack} - {\Omega_{pyr}\lbrack {{2 \cdot ( {{{{sgn}( {i - i_{0}} )} \cdot q_{0,i}} + \frac{l}{2} + {{{i - i_{0}}} \cdot p} + s_{i}} )},\ {2 \cdot ( {q_{0,j} + \frac{l}{2}} )},\ r} \rbrack} + {\Omega_{pyr}\lbrack {{2 \cdot ( {{{{sgn}( {i - i_{0}} )} \cdot q_{0,i}} + \frac{l}{2} + {{{i - i_{0}}} \cdot p}} )},\ {2 \cdot ( {{- q_{0,j}} + \frac{l}{2}} )},\ r} \rbrack} - {\Omega_{pyr}\lbrack {{2 \cdot ( {{{{sgn}( {i - i_{0}} )} \cdot q_{0,i}} - \frac{l}{2} + {{{i - i_{0}}} \cdot p} + s_{i}} )},\ {2 \cdot ( {{- q_{0,j}} + \frac{l}{2}} )},\ r} \rbrack}} \}.}}$

For i=i₀ and j≠j₀ and |q_(0,i)|≤l/2 it is

$\Omega_{i,j} = {( \frac{1}{4} ) \cdot {\{ {{\Omega_{pyr}\lbrack {{2 \cdot ( {q_{0,i} + \frac{l}{2}} )},{2 \cdot ( {{{{sgn}( {j - j_{0}} )} \cdot q_{0,j}} + \frac{l}{2} + {{{j - j_{0}}} \cdot p}} )},r} \rbrack} - {\Omega_{pyr}\lbrack {{2 \cdot ( {q_{0,i} + \frac{l}{2}} )},{2 \cdot ( {{{{sgn}( {j - j_{0}} )} \cdot q_{0,j}} - \frac{l}{2} + {{{j - j_{0}}} \cdot p} + s_{j}} )},\ r} \rbrack} + {\Omega_{pyr}\lbrack {{2 \cdot ( {{- q_{0,i}} + \frac{l}{2}} )},{2 \cdot ( {{{{sgn}( {j - j_{0}} )} \cdot q_{0,j}} + \frac{l}{2} + {{{j - j_{0}}} \cdot p}} )},r} \rbrack} - {\Omega_{pyr}\lbrack {{2 \cdot ( {{- q_{0,i}} + \frac{l}{2}} )},{2 \cdot ( {{{{sgn}( {j - j_{0}} )} \cdot q_{0,j}} - \frac{l}{2} + {{{j - j_{0}}} \cdot p} + s_{j}} )},r} \rbrack}} \}.}}$

The solid angle modifies the linear decay described by equations

*

so that the reduction of light intensities with distance from theclosest pixel is faster than linear. The closest pixel of the imagesensor to the light emitter can still be determined by, for example,identifying the pixel that records the greatest amount of lightintensity. It is also possible to identify the closest pixel bydetermining the middle between pixel pairs that record the closest matchof light intensities from all pixels and that are located on oppositesides of the closest pixel. Such pixel pairs are, e.g., w_(i) _(0+m)_(,j) ₀ and w_(i) _(0−m) _(,j) ₀ . (for m=1 to i_max) or w_(i) ₀ _(,j)_(0+n) and w_(i) ₀ _(,j) _(0−n) (for n=1 to j_max). The closest point inthe plane of the image sensor can be determined to a sub-pixel accuracyby determining q_(0,i) and q_(0,j) (which are the sub-pixel distances ofthe closest point in the plane of the image sensor to the center of theclosest pixel) by taking into account the differences in recorded lightof pixel pairs such as w_(i) _(0+m) _(,j) ₀ and w_(i) _(0−m) _(,j) ₀(for m=1 to i_max) or w_(i) ₀ _(,j) _(0+n) and w_(i) ₀ _(,j) _(0−n) (forn=1 to j_max). An example calculation is also shown below. The measuredlight intensity values w_(i,j) are discrete values so that theidentification of i_max or j_max may be still possible with a highdegree of accuracy in the real world and these values may be used todetermine the distance r from the plane of the image sensor. It shouldbe added that Ω_(i,j) depends also on r so that r cannot be easilydetermined analytically for any given pixels that receive light from thelight emitter. However, it is still possible to determine r in anumerical way from the measured w_(i,j) for any given pixel thatreceives light assuming that the closest pixel and w₀ has beendetermined. It is also possible to use equation

*

to calculate light profiles for different values of r and then fit themeasured light profiles to the calculated light profiles to determine rto a pre-determined accuracy.

It may be added that a more accurate expression for w_(i,j) may requirefurther correction factors compared to equation

**

which take into account, e.g., light recording efficiency of pixels fordifferent incident angles of light. However, such correction factorswould be universal for all pixels and can easily be incorporated inequation

**

. It may also possible to incorporate measured correction factors inequation

**

. The equation

**

also assumes that the light emitter emits light in all directionshomogeneously (i.e., light is emitted in all directions with the sameintensity). It is also possible to take into account inhomogeneous lightemission by using the measured light intensity values from a pluralityof pixels and fit them to a known inhomogeneous light emission profileof the light emitter.

Assuming that the light from the light emitter is at least recorded bynine pixels (one central pixel and the next neighbor pixels), andassuming |q_(0,i) ₀ |<l/2 and |q_(0,j) ₀ |<l/2 (which is a goodapproximation for thin walls) then w₀ may be determined from w_(i) ₀_(,j) ₀ by using

w ₀ =w _(i) ₀ _(,j) ₀ /(l ²·Ω_(i) ₀ _(,j) ₀ ).

q_(0,j) can be determined for example from w_(0,1) and w_(0,−1) andq_(0,i) from w_(1,0) and w_(−1,0). With the approximation |q_(0,i) ₀|<l/2 and |q_(0,j) ₀ |<l/2 and assuming for simplicity that i₀=0 andj₀=0, it follows:

$\begin{matrix}{q_{0,j} = {( {h\text{/}2l} ) \cdot ( {a_{0,{- 1}} - a_{0,1}} )}} \\{= {{( {{h \cdot l \cdot \Omega_{0,0}}\text{/}2w_{0,0}} ) \cdot ( {w_{0,{- 1}} - w_{0,1}} )}\mspace{14mu} {and}}}\end{matrix}$ $\begin{matrix}{q_{0,i} = {( {h\text{/}2l} ) \cdot ( {a_{{- 1},0} - a_{1,0}} )}} \\{= {( {{h \cdot l \cdot \Omega_{0,0}}\text{/}2w_{0,0}} ) \cdot {( {w_{{- 1},0} - w_{1,0}} ).}}}\end{matrix}$

It is thus possible to determine the position of a single light emitterin two dimensions to a sub-pixel accuracy after r has been determinedwith a high accuracy from, e.g., identification of i_max (using formulaabove). The sub-pixel distances between the closest point in the imagesensor plane to the light emitter and the center of the closest pixel,q_(0,i) and q_(0,j), may also be determined from the difference of otherpixel pairs such as w_(0,−2)−w_(0,2) and w_(−2,0)−w_(2,0).

The distance r can also be determined in a different way from any one ofthe measurements w_(1,0), w_(0,1), or w_(1,1) using the equations

*

and

**

. For w_(1,0), it follows:

$\begin{matrix}{w_{1,0} = {w_{0} \cdot \Omega_{1,0} \cdot \lbrack {l - {h \cdot \frac{( {q_{0,i} - \frac{l}{2} + p} )}{r - h}}} \rbrack \cdot l}} \\{= {\lbrack {w_{0,0} \cdot \frac{\Omega_{1,0}}{\Omega_{0,0}} \cdot l} \rbrack \cdot \lbrack {l - {h \cdot \frac{( {q_{0,i} - \frac{l}{2} + p} )}{r - h}}} \rbrack}} \\{= {\lbrack {w_{0,0} \cdot \frac{\Omega_{1,0}}{\Omega_{0,0}} \cdot l} \rbrack \cdot {\lbrack {l - {h \cdot \frac{\begin{pmatrix}{( {h \cdot l \cdot \frac{\Omega_{0,0}}{2w_{0,0}}} ) \cdot} \\{( {w_{{- 1},0} - w_{1,0}} ) - \frac{l}{2} + p}\end{pmatrix}}{r - h}}} \rbrack.}}}\end{matrix}$

The left side is a measurable quantity and the right side depends in acomplicated way on the distance r and known or measurable quantities:the dependency is not only through the r appearing in the equation butalso through the solid angles which depend also on r. It is analyticallydifficult, or improbable, to invert the function and express r in termsof known or measurable quantities but the inversion is possiblenumerically and can be used to determine r from known quantities andquantities that have been measured (after they have been measured).

Thus, it is also possible to determine the distance r of the lightemitter from the image sensor with a sub-pixel accuracy using themeasurements w_(0,0), w_(1,0), and w_(−1,0). It is also possible to usemeasurements from pixels that are further away from the closest pixel todetermine r. It is also possible to determine r in multiple differentways and calculate r with a higher accuracy using statistical methodssuch as averaging.

To determine the brightness of the light emitter, w₀ may be calculatedusing, e.g., the measurement w_(i) ₀ _(,j) ₀ and relate w₀ to thebrightness. This can be achieved based on theoretical calculations(taking into account light measurement efficiency of the pixels andbrightness of the single light emitter) or by calibration with knownlight emitters at known distances from the image sensor. Absorption oflight travelling from the light emitter through a medium to the pixelsmay also be taken into account by adding a correction factor to theequation

**

. The correction factor may depend on the distance that the lighttravels through the medium and may decay exponentially with thisdistance.

In any case, equations

*

and

**

allow for the determination of the precise position of the single lightemitter in the object space and with additional information also thebrightness of the single light emitter. Usually, many pixels will recordlight from the single light emitter and measurements from other pixelscan be used to determine the position and brightness with a higheraccuracy by, e.g., determining the average value of r, average(r). In anexample, light from the light emitter may be measured by 10,000 pixels(in an area of 100×100 pixels) and r may be determined for all 100pixels of a row of the closest pixel (50 to −i-direction and 50 toi-direction) and for all 100 pixels of a column of the closest pixel.The 200 values for r can then be averaged to obtain a more accuratevalue of r. Similarly, the closest pixel (or the two or four closestpixels if the light emitter is above a boundary of a pixel) may bedetermined not only by identifying the pixel that records the maximumlight intensity but also by extrapolating the increase of recorded lightintensity from pixels close to the closest pixel to determine the columnand row containing i₀ and j₀ and thus i₀ and j₀ themselves. It is alsopossible to use measured values of pixels that are located outside therow or column of the closest pixel to determine the parameters of thelight emitter with a high accuracy.

Determining Positions and/or Brightness of Multiple Light Emitters

In some embodiments, a plurality of light emitters is present in theobject space, and some of the light emitters may have overlapping lightprofiles. Using prior systems/methods, it may be difficult to determinethe central or closest pixel for each light emitter (and thereby thedistance r), reducing the accuracy by which positions of the lightemitters are determined. By implementing the systems disclosed herein,an accurate determination of multiple light emitters is enabled, evenwhen these light emitters have overlapping light profiles. For more thana single light emitter in object space, the following exemplary methodscan be performed to determine the position and brightness, sequentially,for each one of the light emitters present in the object space that isobserved.

In one embodiment, a method for determining the position of lightemitters includes the following steps:

-   -   1. Determining one pixel (or two or four) at i₀, j₀ at which a        local maximum of light intensity is measured from a light        emitter.    -   2. Calculating the distance r of the light emitter from the        image sensor and identifying the point of the image sensor that        is closest to the light emitter, for example, by using equations        *        and        **        , optionally further determining the brightness of the light        emitter using further input.    -   3. Subtracting the light profile of the light emitter from the        measured light intensities of the image sensor, for example, by        using equations        *        and        **        .    -   4. Repeating steps 1 to 3 for the next local maximum of light        intensity to determine all parameters for the next light emitter        until no further local maximum can be determined.    -   5. Constructing a complete image of the three-dimensional        observation space with all its light emitters using the        determined parameters for the light emitters. This image may be        used for automated analysis or for creating an image for        displaying to a user.

Regarding step 2, different light emitter configurations may bedistinguished. In one instance, it may be possible that each pixelrecords only light from one light emitter, that is, there is no overlapof light from two different light emitters at one pixel. In such a case,the light emitters can be treated separately as single light emitters.

In another instance, there may be an intersection of the set of pixelsrecording light from one light emitter and the set of pixels recordinglight from another light emitter, but the intersection is relativelysmall so that it can be identified and the parameters for the two lightemitters can be determined from pixels that record light from only oneof the two light emitters (i.e., pixels not in the intersection).

In some instances, the intersection of sets of pixels recording lightfrom two different light emitters is large; that is, it is a substantialportion of two sets, and the two sets have a similar size. This meansthat the two light emitters are close to each other (and have a similardistance r to the image sensor). Some exemplary diagrams illustratingparticular examples of such instances are shown in FIGS. 5-8. Each ofFIGS. 5-8 illustrate a side view of two light emitters and correspondingcones of light recorded by pixels of the image sensor and the resultingcombined light intensity profiles of the two light emitters. Thecombined light intensity profiles include three regions: region oneconsists of the pixels that record light only from the first lightemitter, region two consists of pixels that record light from the firstand second light emitters, and region three records light from only thesecond light emitter. For simplicity, all light intensity illustrationsassume that the solid angle is constant for all pixels so that thechanges in light intensity are linear, as given by the equations

**

for the pixel area that records light.

For example, FIG. 5 illustrates a first light emitter 132 that is near asecond light emitter 134, and each light emitter 132, 134 is shown witha corresponding cone of light 136, 138. Additionally, the combined lightintensity profile 140 is shown, as determined from the photoactivatedpixels receiving light from the light emitters 132, 134. For simplicity,it is assumed that the two light emitters 132, 134 are located along arow of pixels of the image sensor and the light intensities are shownfor this row of pixels. This applies also to the following illustratedexamples. As provided above, the light intensity profile 140 includes afirst region 142, a second region 144, and a third region 146. The firstregion 142 corresponds to a light intensity profile recorded by pixelsreceiving light only from the first light emitter 132. The second region144 corresponds to the combined light intensity profile recorded bypixels receiving light from both the first 132 and second 134 lightemitters. The third region 146 corresponds to a light intensity profilerecorded by pixels receiving light only from the second light emitter132. In the example of FIG. 5, the parameters for the two light emitters132, 134 can be determined from regions one and three, respectively.

FIG. 6 illustrates an example where the proximity of two light emitters148, 150 create a combined light intensity profile 152 where region two154 becomes large. It is still possible to identify the first 156 andthird 158 region to determine the parameters for the two light emitters148, 150, but it is also possible to determine the parameters fromregion two 154 by assuming that the two light emitters 148, 150 areequally bright (otherwise, there would not be a horizontal line 160, buta tilted line, defining the apex of the second region 154) at the samedistance (otherwise, the first region 156 and the third region 158 wouldnot have the same size) and at positions over the image sensor that areidentified by the beginning and end of the horizontal line 160. If theforegoing assumptions are not met, it may still be possible to determinethe parameters for each light emitter using a combination of methodsteps disclosed herein.

In another example, illustrated in FIG. 7, the intersection of sets ofpixels recording light from two different light emitters 162, 164 islarge (i.e., the second region 168 of light intensity profile 166 islarge), and the first 170 and third 172 regions have different sizes.This means that the two light emitters 162, 164 have closest pixels thatare close to each other but are also located at different distances tothe image sensor.

It should be appreciated that in some embodiments, the total lightintensity measured by the image sensor from a light emitter can besimilar and may not depend much on the distance r of the light emitter.One reason for this may be that the angle of the light cone measured bythe image sensor remains the same (i.e., does not increase when thelight emitter gets closer to the image sensor, in contrast to aconventional lens-based system). From this follows that light from alight emitter closer to the image sensor is received by less pixels butwill be recorded in the less pixels with a larger intensity. In contrastto this, a light emitter that is far away from the image sensor is seenby more pixels, but each pixel records a smaller amount of light fromthe light emitter.

Accordingly, assuming that the image sensor receives an equivalent totallight intensity from each light emitter, the total light intensityassociated with the light emitter farther away from the image sensor isspread out among a larger number of pixels compared to the light emittercloser to the image sensor. With continued reference to FIG. 7, thisresult can be exemplified by the disparate slopes and peaks of lightintensity shown in the combined light intensity profile 166. Each pixelreceiving light from the light emitter 162 farther away from the imagesensor registers a light intensity that is incrementally different(greater or lesser) than a neighboring pixel, whereas each pixelreceiving light from the light emitter 164 closer to the image sensorregisters a light intensity that is (comparatively) much different(greater or lesser) than a neighboring pixel. This is because, as statedabove, the sum of light intensities for all pixels receiving light fromeach light emitter 162, 164 is equal, and spreading a defined lightintensity over a larger number of pixels results in an average lightintensity per pixel that is less than the average light intensity perpixel when spreading the same defined light intensity over a smallernumber of pixels.

As shown in FIG. 7, it is possible to first determine the parameters forthe light emitter 162 that has a larger distance to the image sensor andthen subtract the profile of this light emitter 162 from the measuredvalues. Following this, the parameters for the closer light emitter 164can be determined. However, it is also possible to first determine theparameters from the light emitter 164 that is closer to the image sensorand subtract its light profile from the measured values. The secondmethod may, in some instances, be better because the closer lightemitter 164 is more acutely defined (less pixels have to be taken intoaccount and the light intensities are larger), and the local maximum canbe better identified. So, it may be generally better to identify firstthe light emitters that are close to the image sensor. In such aprocedure, it should be noted that generally for a single light emitter,the absolute value of the slope in an i-direction is the same as theabsolute value in the −i-direction (but the sign is different). Thus,calculating half of the difference of the absolute value of the slope inthe i-direction and the −i-direction gives an approximation of a trueslope of the closer light emitter without a contribution from the lightemitter that is further away (this assumes that the contribution of thelight emitter farther away to the slope is constant).

Referring now to FIG. 8—which is a special case because the lightemitters 174, 176 are on the same perpendicular line in thetwo-dimensional cross-section shown in FIG. 8 (as determined by thecombined light intensity profile 178 having a single peak centered onthe second region 180 and flanked by equal, but opposite, first 182 andthird 184 regions)—identifying the different slopes in i-direction andthe −i-direction indicates that two light emitters 174, 176 are presentthat have the same closest pixel but different distances to the imagesensor. In such an instance, it may be better to first determine theparameters for the light emitter 174 that is further away from the imagesensor using the measurements of pixels that record only light from thislight emitter, subtract the light profile of this light emitter, andthen determine the parameters for the closer light emitter 176 using thecorrected measurement values.

In some embodiments, particularly those similar to FIG. 8, the lightintensity profiles for each of the overlapping light emitters 174, 176may be determined, as above, where measurements from pixels in the first182 and/or third 184 regions are used to determine the position of thefarther light emitter 174. The position of this light emitter 174 canthen be used to calculate a theoretical light intensity profile for thelight emitter, which values can be subtracted from the combined lightintensity profile 178 to yield the light intensity profile for thecloser light emitter 176.

It should be appreciated that when the two light emitters are getting asclose as the pixel pitch of the image sensor, they may not be resolvedor differentiated from each other because the resolution of the twolight emitters is less than the resolution of a single light emitter. Itmay be possible to determine that two light emitters are present—owingto the combined light intensity received by pixels being double that ofa single light emitter—but it would be involved to accurately determinethe position of each light emitter.

Three or more light emitters with overlapping light profiles can beidentified by determining the different regions that record light fromone, two, and three light emitters and using them to determine theparameters for each one of the individual light emitters. Similarly,four or more light emitters can be identified by determining thedifferent light regions that record light from each of the four or morelight emitters and using them to determine the parameters for each oneof the individual light emitters.

If there is a line of light emitters that are so close that the distancebetween the light emitters cannot be resolved, it is possible to locatethe line of light emitters in object space in the following way: theline of light emitters can be identified by a line of maximum lightintensity measured by pixels of the image sensor (light intensity islower perpendicularly to the line) and for the distance information,also the values of pixels that are located perpendicularly to the lineof maximum light intensity may be used.

Calculating the Resolution of an Exemplary Image Sensor

In some embodiments, the physical properties of the image sensorassembly (i.e., of the image sensor and the associated structure) can beused to determine a resolution for the corresponding assembly. This caninclude, for example, calculating the parallel and/or perpendicularresolution of an image sensor assembly.

In one or more embodiments, the parallel resolution can be calculated.For exemplary purposes, the parallel resolution can be calculated basedon FIG. 6. For this, the two light emitters can be assumed to be veryclose up to the pixel pitch, p. In this case, it is possible todetermine that there are two different, but close, light emitters ofsimilar brightness at similar distance to the image sensor (otherwisethe light intensity would not be equal for a line of pixels). The numberof pixels that record maximal light intensity can be used to determinethe parallel distance between the two light emitters and this can bedone up to p. Therefore, the parallel resolution is equal to p. Thedistance of the two light emitters can still be determined from, e.g.,the number of pixels recording light from the two light emitters or fromthe slope of the light intensity in the i, −i, j, and −j-direction. Itshould be noted that for a single light emitter, it can be trivial toobtain a sub-pixel-pitch resolution by taking into account differentmeasurements in the i- and −i-directions and in the j- and −j-directions(see above). Such calculations are also possible for two light emitters,but it can become more computationally complex and preferably furtherinterfering light emitters (which can cause different measurements inthe plus and minus directions) are absent so that the parallelresolution of two light emitters can be taken to be approximately p.

In one or more embodiments, the perpendicular resolution can becalculated. For exemplary purposes, the perpendicular resolution can becalculated based, for example, FIG. 8. As the two light emitters aregetting closer and closer, the number of pixels recording only lightfrom the light emitter will become smaller and smaller. The light coneof both light emitters (i.e., the light recorded by pixels in onedirection) has the same angle α_max, which is tan(α_max)=l/h. Adifference D of how far the light cones can be seen by the image sensoris equal to the perpendicular distance between the light emitters times(l/h) because

D=D2−D1=(r+Δ)·tan(a_max)−r·tan(a_max)=Δ·tan(a_max)=Δ·(1/h).

If the minimal value for D is taken to be p (i.e., D>p) so that at leastone line of pixels in all four directions can be identified thatreceives only light from the light emitter that is further away, thenthe perpendicular resolution is p·(h/l) (because Δ·(l/h)=D>p).Therefore, the perpendicular resolution of two light emitters can betaken to be p·(h/l).

In the following, two example applications will be described in moredetail. In a first exemplary application, a relatively large objectspace can be monitored for rare light emitters (as may be the case forcounting applications in flow cytometry or other, similar tasks). Forthis, the image sensor is relatively large and has relatively largepixels: p=10 μm, l=9 μm, the image sensor chip is 30 mm×30 mm—giving aresolution of 9 mp (mega pixels). The object space is assumed to be aslarge as the image sensor chip and the perpendicular size is limited tobe between 1 mm and 10 mm. Some or all of these values may also becompletely different. However, for such values, h=20 μm is a sensiblevalue.

From this follows that light from a single light emitter as close aspossible will be recorded by pixels inside a square of 0.9 mm sidelength, that is, by roughly 8,000 pixels. Light from a single emitter asfar away as possible will be recorded by pixels inside a square of 9 mmside length, that is, by roughly 0.8 million pixels. In someembodiments, it is possible to increase the number of pixels that recordlight from the light emitter by decreasing h.

In the second application, a high resolution will be achieved as may bedone for microscopic applications. For this, the image sensor isrelatively small and has small pixels: p=1.1 μm, l=1 μm, the imagesensor chip is 5.5 mm×5.5 mm, giving a resolution of 25 mp (megapixels). The object space is assumed to be as large as the image sensorchip and the perpendicular size is limited to be between 20 μm and 50μm. For such values, h=2 μm is a sensible value. From this follows thatlight from a single light emitter that is as close as possible to theimage sensor will be recorded by pixels inside a square of 20 μm sidelength, that is, by roughly 320 pixels. Light from a single emitter asfar away as possible will be recorded by pixels inside a square of 50 μmside length, that is, by roughly 2,000 pixels.

It should be appreciated that, in general, the resolution of the imagesensor assembly can be adjustable at least within a directionperpendicular to the surface of the image sensor based on a combinationof one or more of the pixel pitch of the image sensor, the height of thestructure, and the inverse of the side length of a light-sensitive areaof the pixels.

Additional Exemplary Image Sensor Assemblies

The image sensor assembly of FIGS. 2-4 is an example of a more generalconcept that provides wells or a structure that define a certain fieldof view for each pixel. It may also be possible to limit the field ofview to light that is perpendicular to the image sensor (i.e., the fieldof view of each pixel is as large as the pixel itself and has a zeroopening angle), but this would reduce the amount of light that isrecorded from each light emitter, especially the ones that are furtheraway from the image sensor, and would not make it possible to determinethe distance of the light emitter from the image sensor (i.e., notwithout an additional image sensor that is perpendicular to the imagesensor). Further, some of the advantages disclosed above would bereduced and/or eliminated.

In an additional embodiment illustrated in FIGS. 9 and 10, an imagesensor assembly 186 can include an image sensor 114 comprising an arrayof pixels 120 arranged in rows and columns. The image sensor 114 can beassociated with a structure 188 associated with, and extending a heightaway from, the surface of the image sensor 114. The structure 188 caninclude a plurality of walls 190 disposed on the pixel boundaries ofeach pixel (not shown) or a set of pixels 192. For example, as shown inFIG. 9, the walls 190 can surround a group of neighboring pixels, suchas every group of 3×3 pixels. It should be appreciated that although a3×3 group of pixels is illustrated, any number of pixels can comprise agroup of pixels. For example, the wall 190 can surround a 1×2, 2×1, 2×2,2×3, 3×2, 3×4, 4×3, 4×4, or larger group of neighboring pixels accordingto a desired resolution or application.

In the cross-section illustrated in FIG. 10, the walls 190 can include awell-defined horizontal structure 194 on top of the walls 190, forming aT-profile in cross-section. Such walls 190 and horizontal structure 194may be used to further restrict the angle of view of the pixels withoutrequiring an increase in the height of the wall 190. For example, it maybe inefficient to manufacture walls that are ten times higher than thepixel pitch of the image sensor. However, such a wall height may bedesired to allow for monitoring light emitters that are quite far awayfrom the image sensor. Instead of using the desired high walls, it maybe better or preferable to use walls that are, for example, as high asthe pixel pitch of the image sensor and which have a horizontalstructure on top that further reduces the opening. As a result, theopening angle of each pixel is reduced, and therefore, each lightemitter will be seen by less pixels, which again allows for an easierdistinction of light emitters that are far away from the sensor.

The light profile of such walls may be determined analytically, too.Compared to walls without a horizontal structure, the light profile ofwalls with a T-profile (in cross-section) may be changed in such a waythat the difference in light intensity recorded by the closest pixel andcertain pixels close to it is larger.

With reference to the exemplary embodiment of FIGS. 9 and 10, the lightprofile of a light emitter 122 can be calculated for such a wallconfiguration. It follows for the three different pixels per group ofpixels in one direction: for the left pixel of each group of threepixels in the positive i-direction, it is

a_(L_(i)) = p − D/2 − s_(L_(i))  with${s_{L_{i}} = {{h \cdot ( {q_{L_{0}} + \frac{p}{2} + \frac{D}{2} + ( {{3i} - 1} )} ) \cdot p}\text{/}( {r - h} )}},$

with D being the thickness of the horizontal structure (which issymmetrically placed on top of the wall), for q_(L) ₀ ≤p−D/2 and s_(L)_(i) being greater than −5p/2 and less than p/2. Here, the closest pointof the image sensor (not counting the walls) is assumed to be in thecenter of a left pixel and the index i identifies the number of pixelgroups starting from the closest pixel group (and not the number ofpixels). For simplicity, it is assumed that the complete pixel area isable to record light (i.e., l=p).

For the center pixel of each group of three pixels in the positivei-direction, it is

a _(c) _(i) =p−s _(c) _(i) with

s _(c) _(i) =h·(q _(L) ₀ +p/2+D/2+(3i−1)·p)/(r−h)−(p−D/2),

for h·(q _(L) ₀ +p/2+D/2+(3i−1)·p)/(r−h)≥p−D/2 and

for h·(q _(L) ₀ +p/2+D/2+(3i−1)·p)/(r−h)−(p−D/2)≤p.

For the right pixel of each group of three pixels in the positivei-direction, it is

a _(R) _(i) =p−s _(R) _(i) with

s _(R) _(i) =h·(q _(L) ₀ +p/2+D/2+(3i−1)·p)/(r−h)−(p−D/2)−p,

for h·(q _(L) ₀ +p/2+D/2+(3i−1)·p)/(r−h)≥p−D/2 and

for h·(q _(L) ₀ +p/2+D/2+(3i−1)·p)/(r−h)−2p+D/2≤p−D/2.

For such an embodiment, the light profile may be better able to identifythe closest pixel because the first left pixel (i.e., the left pixel ofthe first group of three pixels in the i-direction starting from thegroup of the closest pixel) already has a very much reduced lightintensity because the horizontal structure 194 casts a relatively largeshadow on this pixel. In an example, D may be equal to l (so that eachpixel to the left and right of the wall has an area of l/2 in onedirection that is covered by the horizontal structure), and a lightemitter that is very far away and located above a left pixel of a groupof three pixels will be recorded with full intensity by the center pixelof the closest group, with slightly more or slightly less than half ofthe full intensity by the left pixel (which is the closest pixel) andwith more than half of the full intensity by the right pixel (becauselight can be measured in an area that is directly under the horizontalstructure). This light profile allows for the determination of theclosest pixel of the group.

Furthermore, the closest pixel can be identified by the light intensityof the next left pixel to the right which records less than half of thefull intensity while the center pixel of this group records the fullintensity and the right pixel of this group records more than half ofthe full intensity. The next groups of three pixels to the right (e.g.,in the positive i-direction) will show a light profile in which thelight intensity recorded by the left pixels of each group will quicklybe reduced to zero while the center and the right pixel of each groupwill be reduced to zero much slower (the right pixel being the lastpixel that records any light from the light emitter). The faster decayof the light intensity of the left pixels going to the positivei-direction and the equally faster decay of the light intensity of theright pixels going to the negative i-direction may allow for an easieridentification of the closest pixel for light emitters that are far awayfrom the image sensor.

It is also possible to calculate the distance r from the image sensorfor each one of the three pixel types separately (e.g., by determiningthe furthest pixel of each type—left pixels, center pixels, and rightpixels—that still records light) to confirm that the values for r areconsistent. Alternatively, it may be possible select only one pixel typefor calculating the r value based on the range of r.

In further embodiments, the wells created by the structure associatedwith the image sensor may have a different shape and/or cross sectionother than that disclosed above. For example, the well structure can bearcuate (e.g., round like a circle) or polygonal (e.g., hexagonal). Thewells may also be associated with microlenses disposed atop of the wellsto increase the field of view or to focus light onto a light sensitivearea of the pixels.

In some embodiments, the pixels may also have optical filters to limitthe light recorded at the pixels to one or more certain wavelengthranges. The optical filter material may be located inside the wells,optionally up to the height of the walls. In such cases, the abovecalculations may be modified to take into account refraction of light atthe optical filter material. The optical filter material may also belocated above the wells.

For embodiments where the image sensor has one optical filter in allwells, the image sensor may be quickly exchangeable with another imagesensor having a different optical filter to be able to make measurementsin different wavelength ranges.

In one or more embodiments, the image sensor may have different opticalfilter material in different wells. The different optical filtermaterial may be arranged in a color filter array, and the color filterarray may be a Bayer pattern, which is often used for image sensors indigital cameras. Such an image sensor allows for recording of coloredimages but may result in the resolution being lower.

In some embodiments, 2 or more image sensors can be used to record lightfrom the same object space at the same time. The image sensors may,e.g., be positioned in front of the object space, behind it, and/or atsides thereof. The plurality of image sensors may be positioned atdefined angles with respect to each other, such as, e.g., orthogonallyto each other and such that the object space can be illuminated from oneside (e.g., above) with excitation light. Each of the plurality of imagesensors may have a different optical filter material to provide colorimages or they may be used to increase the resolution of the system,especially if the object space is large and contains quite a lot oflight emitters. In some embodiments, a plurality of image sensors ispositioned serially—imaging a different section of object space—to covera larger object space.

The image sensor may also be movable with respect to the object space sothat the object space can be imaged from different points of view inorder to increase resolution. This can be particularly advantageous if alot of light emitters are present in the object space. In an embodiment,the image sensor may be moved by tilting the image sensor with respectto the object space so the field of view of the pixels is changed. In afurther embodiment, the image sensor may be moved along a portion of asphere to record light from the object space from different directions.In still a further embodiment, the image sensor can be translated in az-direction with respect to the object space, effectively growing,shrinking, or translating the object space.

The image sensor can also be used to observe a point (or very smallregion) in the object space. For this, a single converging lens,possibly a Fresnel lens, may be placed on top of the image sensor tocover all wells. The wells may be deep or have a very small field ofview so that only perpendicular light from the converging lens isrecorded by the pixels. The perpendicular light is light that is comingfrom the focal point of the converging lens so that light from otherregions of the object space is effectively filtered out.

An image sensor as described above may also be used for recording lightgoing through the sample (e.g., light that is not emitted by an emitterin the object space). The light going through the sample may becollimated light (perpendicular to the image sensor surface) as it maybe used in projection microscopy. However, in contrast to projectionmicroscopy, the overlapping field of view of different pixels of theimage sensor allow for a determination of a distance of objects from theimage sensor although it may be better to use pixels with a small fieldof view (i.e., relatively high walls that are at least as large as thepixel pitch). Alternatively, the light may be considered as beingemitted by one or more light emitters in the object space and a lightintensity profile may be calculated or measured for the one or morelight emitters in combination with a structure located between the oneor more light emitters and the image sensor. The structure may be opaqueso that it blocks a portion of the light that would otherwise berecorded by pixels of the image sensor. In an example, the structure maybe a very thin square structure with a side length of the pixel pitchand oriented in parallel to the surface of the image sensor. Such alight intensity profile may then be used as a basis to identifyarbitrary shapes of structures blocking light from one or more lightemitters that are farther away from the image sensor than the arbitrarystructures.

In some embodiments, the image sensor assembly includes or is associatedwith a backside illuminated chip for increasing a light sensitive areaof each pixel or each set of pixels. Additionally, or alternatively, theimage sensor assembly can be associated with a transparent materialdisposed in the space between the structure and the image sensor. Insome embodiments, the transparent material is disposed above thestructure and/or between the structure and the image sensor. This can bebeneficial for protecting the image sensor assembly in implementationswhere the sample includes liquids or chemicals that are directlyassociated with the image sensor (e.g., partially submerging the imagesensor assembly in the sample).

Image Sensors Associated with Lens Systems

While the image sensors described above do not necessarily have alimited depth of field like a standard camera system, it may nonethelessbe difficult to identify and/or position distant light emitter if theyare positioned too far away from the image sensor. One reason for thismay be that the change of light intensity recorded by different pixelsfrom a distant light emitter is small, making it difficult to identifythe closest pixel that records the greatest amount of light. Thisresults in a loss of fidelity and/or increased difficulty whendetermining the position of the light emitter with respect to theco-planar directions of the image sensor. In some instances, this caneven result in the inability to determine the exact number of lightemitters within the object space. Even if a closest pixel (e.g., localmaximum of light intensity) can be identified, it may be difficult toconfidently determine the distance of the light emitter from the imagesensor owing to the slow decay of light intensity measured by thesurrounding pixels. An additional complicating factor includessituations where the light emitters have overlapping light profiles tothe extent that the pixels are receiving light from more than one lightemitter.

At least some of the foregoing problems may be addressed by moving theimage sensor into closer proximity with the light emitters. However, insome instances, it may be impractical or difficult to move the imagesensor and/or the light emitters into an operable proximity, and evenwhere it is possible, the size of the object space in the directionperpendicular to the image sensor may still be too limited and/or thesize of the object space in the directions parallel to the image sensormay be too limited. As has been shown above for a few examples, the sizeof the object space (e.g., in a direction perpendicular and/or parallelto the image sensor) may be limited and may be smaller than the volumein which the light emitters are located.

Embodiments of the present disclosure address this and other problems inthe art.

For example, image sensors described herein can be associated with alens system—positioned between the image sensor and the object spacecomprising the light emitters—so that the available object space isshifted away from the image sensor and/or increased in directionperpendicular to the image sensor. Preferably, the addition of a lenssystem creates an image space between the lens system and the imagesensor where real images of the light emitters can be projected. Eachreal image of respective light emitters can then be identified andcounted and/or positioned within the image space, as described above forlight emitters. Because the properties of the lens system and imagesensor (with associated structure) are known, the positions of the realimages within the image space can be used, in some embodiments, tocalculate the actual positions of the light emitters in the objectspace.

In other words, the limitations that have been formulated for the objectspace of embodiments without a lens system, discussed above, can now beapplied to the image space (e.g., restriction of the object space indirection perpendicular to the image sensor can apply now to the imagespace) and the light emitters can be considered as being replaced by thereal images of the light emitters. As a consequence, limitations for theobject space can now be changed depending on the lens system. Thus,using the lens system allows for a more distant positioning of theobject space with respect to the image sensor and also for a larger sizeof the object space especially in direction perpendicular to the imagesensor. Preferably, the lens system is configured to create only realimages of light emitters that are located in the object space and notvirtual images. One reason for this is that real images of lightemitters may be easier to identify in some embodiments because theyrepresent close and localized apparent sources of light. Embodimentswith a lens system may be constructed in the following manner: dependingon, for example, the resolution requirements of the application or theamount of light collection, the image sensor with its pitch and thestructure with its height may be selected and depending on requirementsregarding the object space, a lens system may be selected with itsappropriate optical parameters.

As such, embodiments of the present disclosure additionally enable theidentification, counting, and/or positioning of light emitters within anobject space that may otherwise have been difficult to identify, count,and/or position due to the relative distance between the image sensorand light emitters and/or between the light emitters within the objectspace.

Referring now to FIG. 11, illustrated is a schematic of the image sensor114 of FIGS. 2-4 shown in association with an exemplary lens system 200.As shown, the lens system 200 is positioned between the image sensor 114and the light emitter 202. The lens system 200 acts to functionallydivide the viewable area of the image sensor 114 into an object space204 on one side of the lens system 200 and an image space 206 on anopposing side of the lens system 200. The light emitter 202 is disposedwithin the object space 204 and projects light through the lens system200. In turn, the lens system 200 projects a real image 208 of the lightemitter 202 into the image space 206. The real image 208 can then beidentified by the image sensor 114 (with associated structure), asprovided above. The same or similar methods described above canadditionally be used to calculate a position of the real image 208relative to the image sensor 114.

In some embodiments, the lens system 200 has the effect of a converginglens (e.g., a biconvex lens), which, as shown in FIG. 11, produces areal image 208 of the light emitter at a distance, a′, away from thelens system 200 in the direction of the image space 206. In someembodiments, the light emitter 202 and the real image 208 areequidistant from the lens. That is, the distance, a, of the lightemitter 202 in the direction of the object space 204 is the same as thedistance, a′, of the real image 208 in the direction of the image space206 (i.e., a=a′). In some embodiments, the lens system 200 comprises oneor more lenses such that a≠a′.

When considering convergent lens systems described by a focal length f,the Gaussian lens equation for conjugate planes at a distance a and a′from the lens is:

${\frac{1}{f} = {\frac{1}{a} + \frac{1}{a^{\prime}}}},$

where a describes the distance of the object plane from the lens and a′describes the distance of the real image from the lens. In someembodiments, the Gaussian lens equation may be used to determine theparameters of the lens system 200 for a given image sensor 114.

In the context of FIG. 11 and where the light emitter 202 is positioneda distance, a, away from the lens system 200 such that the focal lengthof the associated lens, f, is less than a, the foregoing provides thatthe light emitter 202 within the object space 204 appears as a realimage 208 in image space 206 a distance of a′ away from the lens system200. To the image sensor 114, the real image 208 appears to be a lightemitter 202 that emits light. As such, the apparent light emitted by thereal image 208 can be recorded by the image sensor 114 and used for anidentification of the apparent light emitter (i.e., the real image 208of the light emitter 202) and a determination of the position of theapparent light emitter. In some embodiments, changes of the lightemission characteristics of the apparent light emitter compared to theisotropic light emission of the actual light emitter can be taken intoaccount when determining their relative positions. However, for countingapplications, such compensation may not be incorporated, as long as thelight emitters can be reliably identified as local maxima in the lightintensity profiles measured by the pixels of the image sensor.

Generally, the lateral magnification of such lens systems is known to beM=a′/a and the angular magnification γ=a/a′. Accordingly, an increase ofthe object space over the image space in a direction perpendicular tothe image sensor (i.e., a>a′) causes an increase of the object spaceover the image space in direction parallel to the image sensor becausethe lateral magnification M is less than 1. Also, the angularmagnification is increased, which may ensure that the light emission ofthe real image is wide enough so that every pixel of the image sensorthat sees the real image (i.e., the real image is in the field of viewof the pixel) also receives light from the real image. It should benoted, however, that the intensity of light emitted by the real imagedecreases as the angular magnification increases. It is also noteworthythat the lateral and angular magnifications depend on the relativedistances of the light emitter and real image from the lens system andthus change when this ratio changes.

As a simple, non-limiting example, an image space may be restricted inthe direction perpendicular to the lens (and thus, to the image sensor)to a distance z′ away from the lens in the following way: f<z′ andz′<2·f. According to the Gaussian lens equation, this allows recordingof a light emitter in the distance z from the lens with 2·f<z and z<∞.This assumes that the image sensor can identify images of light emittersin the complete image space. For example, f may be selected within therange of 1 mm-2 mm for an image sensor having a pixel pitch of 1 μm anda wall height of 1 μm-2 μm.

In some embodiments, the object space may have a certain extension, forexample 25 mm, in a direction perpendicular to the image sensor. Forexample, the light emitters in a sample vessel and the real images ofthe light emitters may be limited to:

${( {1 + \frac{1}{10}} ) \cdot f} < {z^{\prime}\mspace{14mu} {and}\mspace{14mu} z^{\prime}} < {( {1 + \frac{2}{10}} ) \cdot {f.}}$

This allows recording of light emitters in the distance z from the lenswith approximately 6·f<z and z<11·f. Therefore, the object space is 50times larger than the image space in the direction perpendicular to theimage sensor. In embodiments where the image space has a perpendicularextension of 0.5 mm, which may be reasonable for a pixel pitch and wallheight of around 1 μm, it follows that for f=5 mm, the object space hasa perpendicular extension of 25 mm as required. Such an exemplarydimensioned object space is sufficient in certain embodiments to monitora sample comprising light emitters disposed within a larger samplevessel, including standard sample vessels, as long as the liquid doesnot extend more than about 2.5 cm in vertical direction. As oneexemplary embodiment using these foregoing values, the lens system canbe mounted 3.5 cm above the liquid surface. When mounted in this way,the lateral magnification is in the range of around 1/10−⅙ and theangular magnification is in the range of around 6-10.

In some embodiments, it may be preferable to have a smaller lateralmagnification, and embodiments of the present disclosure enable theimage space to be between two and sixty times smaller than the objectspace in a direction perpendicular to the image sensor. For example, itis possible to select a lens having a focal length of f=40 mm and tolimit the distance z′ of the real image to:

${{( {1 + \frac{15}{80}} ) \cdot f} < {z^{\prime}\mspace{14mu} {and}\mspace{14mu} z^{\prime}} < {( {1 + \frac{16}{80}} ) \cdot f}},$

which is limiting approximately to 47.5 mm<z′ and z′<48.0 mm. Thisallows recording of light emitters in the distance z from the lens withapproximately 5·f<z and z<5.33·f, which limits z′ approximately to 200mm<z′ and z′<213 mm. Therefore, the object space is around 26 timeslarger than the image space in direction perpendicular to the imagesensor, the lateral magnification is around ¼, and the angularmagnification is around 4.

Although shown as a single lens, it should be appreciated that the lenssystem 200 of FIG. 11, and other lens systems described herein, caninclude one or more lenses. Further, each lens and combination of lensesprovided within disclosed lens systems can be selected in accordancewith the type of lens system (e.g., convergent or telecentric lenssystem), focal lengths of the lenses, size of the object space, distancebetween the image sensor and light emitters, size and resolution of theimage sensor, and/or the number, concentration, and/or intensity oflight emitters within the object space.

For example, a lens system provided in association with the image sensorcan include a first and a second convergent lens system that satisfies:

distance(L ₁ ,L ₂)≥(2·f ₁)+(2·f ₂),

-   -   where        -   L₁ represents the first convergent lens system,        -   L₂ represents the second convergent lens system,        -   f₁ represents the focal length of L₁, and        -   f₂ represents the focal length of L₂.

In such an exemplary embodiment, the real image viewed by the imagesensor can be created by the second convergent lens system. In thisexample, objects that are farther than 2·f₁ away from the firstconvergent lens system have a first real image that is in the rangebetween f₁ and 2·f₁ and also have a second real image generated by thesecond convergent lens system that is in the range between f₂ and 2·f₂from the second convergent lens system. The total magnification can becalculated as the product of the two magnifications of each lens. Itshould be noted that in embodiments having two convergent lens systemswithout a telecentric relationship, further configurations may be used,but care should be taken to avoid the second image being a virtual imageinstead of the preferred real image.

In some embodiments, the lens system is a telecentric lens system. FIG.12 illustrates a schematic of a telecentric lens system 210 thatembodies the foregoing. As shown, the telecentric lens system 210includes a first convergent lens system 212 and a second convergent lenssystem 214. The first convergent lens system 212 has focal length f₁ andthe second convergent lens system has a focal length f₂ such that f₁>f₂.As shown, the first convergent lens system 212 is spaced apart from thesecond convergent lens system 214 at a distance equal to the sum of thefocal lengths of each convergent lens system. The light emitter 216 ispositioned within the object space 204, and the light emitted thereby isprojected through the lens system 210 to create a real image 218 of thelight emitter 216 (i.e., the second image generated by the lens system210) within the image space 206. The real image 218 acts as an apparentlight emitter, allowing the image sensor 114 to record light “emitted”from it. The same or similar methods described above can additionally beused to calculate a position of the real image 218 relative to the imagesensor 114.

It should be appreciated that although the first and second convergentlens systems 212, 214 are illustrated as a single element in theschematic depicted in FIG. 12, the first and/or second convergent lenssystems may have one lens or a plurality of lenses.

Further, in the telecentric lens system illustrated in FIG. 12, thelateral magnification is M=(f₂/f₁), the angular magnification γ=f₁/f₂,and the axial magnification is α=M²=(f₂/f₁)². As can be seen from theseequations, the magnification values are constant for given lens systems(i.e., they do not depend on the distance of the light emitter from thelens system) and the axial magnification, which can be beneficiallyadjusted, increases quadratically with the ratio of the focal lengthswhile the lateral magnification increases linearly.

As an example, a telecentric lens system may include a first lens systemhaving f₁=25 mm, f₂=5 mm, so that α= 1/25, M=⅕, and γ=5. As such, animage space of 0.5 mm perpendicular size (which is appropriate for animage sensor having a pixel pitch of 1 μm and a wall height of 1 μm)will allow for an object space of 12.5 mm perpendicular size. Morespecifically, using the Gaussian lens equation for the first and thesecond lens, it can be calculated that an object space with a range off₁ to (1.5·f₁) (i.e., a perpendicular size of 12.5 mm) from the firstlens will have a final image space in the range of (0.9·f₂) to f₂ (i.e.,a perpendicular size of 0.5 mm). More specifically, the distance s′₁ ofan image of a light emitter at distance s₁ generated by the first lensis

$s_{1}^{\prime} = {f_{1}\text{/}{( {1 - \frac{f_{1}}{s_{1}}} ).}}$

Distances s₁ and s′₁ are with respect to the first lens. The distances′₁ of the image of the first lens with respect to the first lens can beidentified with the distance s₂ of a virtual object from the second lensthrough the following equation: s₂=−s′₁+f₁+f₂. The virtual object at s₂has then an image at the distance s′₂ from the second lens which isgiven by

$s_{2}^{\prime} = {f_{2}\text{/}{( {1 - \frac{f_{2}}{s_{2}}} ).}}$

It can also be calculated that an object space with a range of (1.5·f₁)to (2·f₁) (i.e., again a perpendicular size of 12.5 mm) from the firstlens will have a final image space in the range of (0.8·f₂) to (0.9·f₂)(i.e., again a perpendicular size of 0.5 mm). This confirms that theaxial magnification values are independent of the distance of the lightemitter from the lens system and have the expected value of α= 1/25.Therefore, in an example the object space may be located between 50 mmand 37.5 mm in front of the first lens and the image space may belocated between 4 mm and 4.5 mm behind the second lens. It should benoted that the object space may not arbitrarily extend in this waybecause objects that are too far away from the first lens system canproduce a virtual second image, which should be avoided.

In a further example, f₁ may be equal to f₂ so that the magnificationvalues are all equal to 1. The telecentric lens system may then be usedto shift the space that can be monitored by the image sensor away fromthe image sensor without increasing this space.

It should be appreciated that although specific reference was made toimage sensor 114 of FIGS. 2-4 when discussing the concepts illustratedwithin FIGS. 11 and 12, other image sensors are compatible and can beused with the lens systems described herein to achieve the noted andimplicit benefits associated therewith. For example, the lens systemsdescribed herein can be associated with the image sensor 186 of FIGS. 9and 10 or with any other image sensor described herein.

Apparatuses, Systems and Methods for Identifying a Plurality ofLight-Emitting Droplets

As described above, the systems and methods disclosed herein offersignificant advantages over prior art microscopy systems and methods.The disclosed systems enable the three-dimensional positioning oflight-emitting objects within an object space without moving the sampleor a system of lenses through a series of z-sequences, thereby reducingcost and mechanical/operational complexity of the imaging systems.Additionally, images can be obtained more quickly and with a reduceddigital storage cost (e.g., because an associated computing system onlystores data corresponding to a single image instead of datacorresponding to stacks of images and the spatial relationships betweeneach image of the stack).

With respect to fluorophore-based light emitters, the disclosed systemsand methods beneficially reduce the amount of time fluorophores areexposed to excitation light without sacrificing the precision by whichthe positions of the light emitters are determined within the sample.For example, a single image can be captured and the position of one ormore light emitters determined therefrom.

Additionally, or alternatively, the systems and methods disclosed hereinenable the identification of multiple light emitters within the objectspace of the image sensor. The operable range and breadth of the objectspace can be increased through the addition of an intervening lens, asdescribed, for example, in FIGS. 11 and 12. This can equate to a greateroperating distance between the image sensor and the sample and/or anobject space having a larger operable volume (e.g., increase the width,length, and/or depth of the object space).

Many of the foregoing and other benefits of the disclosed systemsadvantageously enable various practical applications of this technology.For example, there is an outstanding problem in the art of nucleic acidmonitoring and visualization during various implementations of PCR, suchas digital PCR (dPCR). The various implementations of dPCR can be usedto directly quantify and clonally amplify nucleic acid templates,including single or double stranded DNA, cDNA or RNA, with greaterprecision than traditional PCR methods, as known in the art. Instead ofacquiring a single data point in traditional PCR approaches, dPCRseparates a sample into a large number of discrete small-volumereactions. Each of the small-volume reactions is typically associatedwith a reporter molecule (e.g., nucleic acid probes comprising afluorescent dye and a quencher such as TaqMan® probes available fromThermo Fisher Scientific Inc., nucleic acid intercalating dyes such asethidium bromide, nucleic acid groove binding dyes such as Hoechst dyes,nucleic acid stains such as SYBR® Green dyes available from ThermoFisher Scientific Inc., and other fluorescent markers known in the art),and following thermal amplification cycles, the presence andconcentration of amplicons within each small-volume reaction can bedetermined. From these data, and based on the notion that the sample hasbeen distributed randomly (i.e., following a Poisson distribution) intoeach of the small-volume reactions, a precise measurement of thequantity of the target sequence can then be determined.

Droplet PCR or emulsion PCR is a subset of digital PCR. In droplet PCRmethods, a water in oil emulsion is prepared with droplets that containfew or no single target nucleic acid molecules. Statistically, it may beexpected that there is less than one target nucleic acid molecule ineach droplet (e.g., an average of 0.5 or less target nucleic acidmolecules per droplet). The droplets are then thermally cycled in a PCRprocess and after a sufficient number of cycles, the droplets that emitfluorescence are identified as droplets in which at least one targetnucleic acid molecule was present before the PCR cycling. Measuring orcalculating the total number of droplets allows a determination of thetotal number of target nucleic acid molecules that were present in thevolume of aqueous sample used to prepare the droplets.

Thermal cycling of droplets in droplet PCR is typically performed withinan emulsion located in a sample vessel. Due, at least in part, to thedifficulty of standard optical systems to measure droplets in a largevolume (e.g., because the depth of field is too small and many of thedroplets are too blurry to be identified), the measurement of dropletfluorescence is usually spatially divorced from the thermal cycling.That is, prior art systems may force the user to sequentially measurethe fluorescence of droplets. This typically requires moving theemulsion from the sample vessel to a different container of an opticalsystem that is capable of sequentially identifying the dropletsexhibiting fluorescence. This may require moving the emulsion from oneapparatus (e.g., the thermal cycler) to another apparatus (e.g., afluorescence detector) that can align the droplets in a row so that theycan be sequentially investigated. Thus, although theoretically possible,current technologies are inefficient in monitoring the fluorescencevalues of droplets in a sample vessel undergoing active thermal cycling.There is a need for systems and methods that identify fluorescentdroplets in a sample vessel that may also be used for thermal cyclingand/or for systems and methods that identify fluorescent droplets withina sample reservoir and monitor the increase of fluorescence duringthermal cycling, particularly while the sample vessel is associated withthe thermal cycler.

Systems and methods of the present disclosure solve one or more of theforegoing and other problems in the art of microscopic imaging,particularly identifying and/or monitoring light-emitting droplets indroplet PCR or similar. For example, the disclosed apparatuses foridentifying a plurality of light emitting droplets include detectorsystems comprising an image sensor and associated walled-structure.These detector systems can accurately and quickly count the number ofdroplets within the same sample vessel and can be combined with athermal cycler for cycling the emulsion in the sample vessel to identifyand monitor droplet fluorescence while the droplets are in the samplevessel and/or remain associated with a thermal cycler. In someembodiments, a lens system is positioned between the image sensor andthe sample vessel to increase the operable distance, further enablingadditional variations and embodiments.

Referring now to FIG. 13, illustrated is a general schematic of athermal cycling and imaging system 220 for identifying a plurality oflight emitting droplets. As shown, the system 220 includes a samplevessel 222, optionally associated with a thermal cycling unit 224. Thesample vessel 222 is configured in size and shape to hold a plurality ofdroplets and to transmit changes in temperature (e.g., during thermalcycling). In an exemplary embodiment, the sample vessel 222 is amicrofluidics chamber housing an emulsion of small-volume droplets or anindividual well of a multi-well plate or substrate. As used herein, theterm “small volume” when referring to droplet volume is intended to beon the nanoliter to picoliter scale. For example, a small-volume dropletcan have a volume less than 100 nL, less than 10 nL, less than 1 nL,less than 100 pL, less than 10 pL, less than 1 pL, less than 100 fL,less than 10 fL, less than 1 fL, or a volume within a range selectedfrom the foregoing volumes as endpoints (e.g., between 1 fL-1 pL,between 10 pL-100 nL, between 10 nL-10 pL, between 10 nL-100 pL, between1 nL-100 pL, or between 1 nL-10 pL). In an example, the droplets mayhave a diameter between 0.5 μm and 10 μm which results in a volumebetween about 0.065 fL and 0.5236 pL. In a further example, the dropletsmay have a diameter between 2 μm and 5 μm.

The desired contents and droplet volume can be adjusted or selected, asknown in the art. For example, the droplet contents and volume can becontrolled when forming the emulsion. An emulsion can be created byflowing a discontinuous phase fluid into a continuous phase fluid ofopposite polarity. In various exemplary embodiments, a continuous phasecan be can include an oil (e.g., mineral oil, light mineral oil, siliconoil) or a hydrocarbon (e.g., hexane, heptane, octane, nonane, decane,etc.) and the like. The composition of the continuous and discontinuousphases can be selected at the discretion of the practitioner, thoughgenerally, the composition of the various phases can be selected toprovide a suitable emulsion under thermal cycling conditions. As usedherein, the term “suitable emulsion” and equivalents refers to anemulsion that does not substantially degrade or collapse or in which thehydrophilic compartments do not substantially coalesce under thermalcycling conditions. Therefore, in various exemplary embodiments, anemulsion can be suitable for carrying out reactions at varyingtemperatures (e.g., thermocycling), and other conditions (e.g., pH,ionic strength, hybridization conditions), and can contain variousreaction components (e.g., sample nucleic acids, proteins, enzymes,catalysts, co-factors, intermediates, products, by-products, labels,probes, microparticles, etc.).

It should be appreciated that the aqueous droplets may have the samedensity as the oil of the emulsion so that they become more or lessevenly distributed within the volume of the sample vessel. In someembodiments, the droplets may have a lower (or higher density) than theoil so that they have a higher concentration at a defined distance fromthe image sensor (e.g., where the image sensor has the highest effectiveresolution).

In one embodiment, the emulsion is generated such that each dropletcontains PCR reagents. Additionally, each droplet can contain afluorescent marker configured to indicate presence of the target nucleicacid. The fluorescent marker can include nucleic acid probes comprisinga fluorescent dye and a quencher such as TaqMan® probes available fromThermo Fisher Scientific Inc., nucleic acid intercalating dyes such asethidium bromide, nucleic acid groove binding dyes such as Hoechst dyes,nucleic acid stains such as SYBR® Green dyes available from ThermoFisher Scientific Inc., and other fluorescent markers known in the art.In some embodiments, each droplet contains a droplet identifyingfluorescent marker so that the apparatus can identify each droplet anddifferentiate the target-nucleic-acid-containing droplets from thenon-target-nucleic-acid-containing droplets or identify the total numberof droplets.

In some embodiments, each droplet contains a plurality of fluorescentmarkers. The emission wavelengths of each of the plurality offluorescent markers can be different such that different multiplex PCRamplification products can be identified and/or monitored. In anexemplary embodiment, each of the plurality of droplets within anemulsion is formed such that each includes PCR reagents and buffers,polymerase, and one or more fluorescent markers. The concentration ofsample may be varied by the practitioner. However, in some embodiments,each of the plurality of droplets within the emulsion is formed suchthat each includes (probabilistically) at least one copy of at least onetarget nucleic acid. In different embodiments, each of the plurality ofdroplets within the emulsion is formed such that each includes(probabilistically) less than one copy of at least one target nucleicacid.

In another exemplary embodiment, each of the plurality of dropletswithin an emulsion is formed such that each includes PCR reagents andbuffers, polymerase, and at least two fluorescent markers. A firstfluorescent marker can identify a first multiplex PCR amplificationproduct, and a second fluorescent marker can identify a second multiplexPCR amplification product. Again, the concentration of sample may bevaried by the practitioner. However, in some embodiments, each of theplurality of droplets within the emulsion is formed such that eachincludes (probabilistically) more than (or in some instances less than)one copy of at least one target site for multiplex PCR.

The sample vessel can include a multi-well plate or substrate,preferably a high-density multi-well plate or substrate. In oneembodiment, the multi-well plate or substrate includes at least 96 wellsconfigured to hold between 1 μL-200 μL of sample volume. The disclosedsystems are additionally compatible with higher density multi-wellplates, such as plates having 384 wells, 1536 wells, 3456 wells, 6144wells, 9600 wells, or more. The volumes of each well can vary, as knownin the art, but should preferably retain more than at least 100 nL ofsample volume. It should be appreciated that other substrates can beused, such as microcards, microfluidics chambers, and the like—insofaras the droplets within the associated wells or sample vessels are withinthe object space of and viewable by the image sensor.

It should be appreciated that the selection of multi-well plate and/orsubstrate can affect the number and type of detector systems utilizedwithin the thermal cycling and imaging system 220. For example, a singledetector system can be integrated into the thermal cycling and imagingsystem 220 in situations where the entire sample vessel is within theoperable object space of the associated image sensor. In someembodiments, additional detector systems can be permanently or removablyassociated with the thermal cycling and imaging system so that eachsample vessel (or at least the droplets contained therein) fall withinthe operable object space of at least one image sensor.

With continued reference to FIG. 13, the thermal cycling and imagingsystem 220 can include an image sensor 114 and lens system 200, such asthose described above. In some embodiments, the image sensor 114 andlens system 200 are configured in such a way that all droplets in thesample vessel 222 can be identified and counted. Stated another way, theimage sensor 114 and lens system 200 are positioned relative to thesample vessel 222 such that the field of view of the image sensor 114includes the whole cross section of the sample vessel 222 and such thatthe farthest and nearest possible distances from the image sensor 114 atwhich the image sensor 114 can identify light-emitting droplets includesthe position of the sample liquid in the sample vessel 222. In short,all the droplets within the sample vessel 222 are located within theobject space defined by the image sensor 114 and the additional lenssystem 200.

The thermal cycling unit 224 of system 220 is, in some embodiments,optional. When included, the thermal cycling unit 224 can be a standardthermocycler, as known in the art, or it may have components of astandard thermocycler and be disposed opposite the lens system 200 andimage sensor 114 so as to not interfere with the line of sight betweenthe image sensor 114 and the sample vessel 222. In this way, the samplevessel 222 or the droplets contained therein are not required to bemoved for surveillance of fluorescence and the droplets within thesample vessel 222 may be monitored while simultaneously being thermallycycled. Among other benefits, such implementations enable the evaluationof how fluorescence within the droplets increases in real time, whichcan be used to more accurately compute the number of target nucleic acidmolecules present in the droplets. In some embodiments, the lens system200 and the image sensor 114 may be positioned above the sample vessel222 that may be covered with a transparent foil or an oil to preventevaporation during the cycling.

Referring now to FIG. 14, illustrated is an exemplary embodiment of asample vessel 222 having a plurality of light emitting droplets 228identified by an image sensor 114. As shown, the sample vessel 222includes an emulsion of droplets during and/or following thermallycycling the emulsion. Fluorescent probes within each droplet act todifferentiate subpopulations of the emulsion droplets by identifyingthose light-emitting droplets 228 having the target nucleic aciddisposed and amplified therein. The remaining,non-target-nucleic-acid-containing droplets 226 can be clear, as shown,or in some embodiments, these droplets can include a differentfluorescent marker so that the image sensor can positively identify eachdroplet within the emulsion and further differentiate the target nucleicacid containing droplets from the rest of the droplets.

Systems for identifying and/or monitoring light-emitting droplets canadditionally include one or more light sources. As shown in FIG. 14, thesystem may include a light source 225 for illuminating the sample vessel222. The light source 225 may be located next to the image sensor 114and, when present, the additional lens system. The image sensor 114 maybe located above the center of the sample vessel 222 (as shown) with thelight source 225 positioned adjacent to the image sensor 114 so that itcan illuminate the entire sample vessel 222, or at least the dropletcontaining portion of the sample vessel.

In one embodiment, two or more light sources may be located around thecentrally positioned detector system. Additionally, or alternatively, asemi-transparent mirror or a dichroic mirror may be used to directexcitation light centrally from above the sample vessel into the sampleand to provide emission light to the detector system. In such examples,the excitation light may be reflected by the mirror down into the sampleand the emission light may pass through the mirror up to the to theimage sensor 114 or it may be the other way around (i.e., excitationlight may pass through the mirror and the emission light may bereflected by the mirror).

Accordingly, the disclosed systems can include one or more light sources225 to illuminate the sample vessel 222. The light sources can be, forexample, a white light or a light of a defined wavelength. Inembodiments where the light-emitting droplets include a fluorophore (ora plurality of fluorophores), the light source can include one or morefluorophore excitation light sources. For example, a light enginecomprising multiple LEDs each configured to emit an excitationwavelength (or range of wavelengths) for exciting fluorophores withinthe sample vessel can be positioned adjacent to the image sensor oraround an optical axis of the lens system. Additionally, oralternatively, optical filters that filter the excitation and emissionlight, such as a multi-position dichroic filter wheel and/or amulti-position emission filter wheel can be associated with the imagesensor, as appropriate.

As a general method of operation, a fluorophore excitation source can beautomatically or manually directed to provide multiple bandwidths oflight ranging from violet (e.g., 380 nm) to near infrared (e.g., atleast 700 nm) and are designed to excite fluorophores, such as, forexample, cyan fluorescent protein (CFP) and Far Red (i.e., near-IR)fluorophores. Example LED bandwidths with appropriate excitation filters(e.g., as selected via an excitation filter wheel) can include, but arenot limited to, Violet (380-410 nm LED & 386/23 nm excitation filter),Blue (420-455 nm LED & 438/24 nm excitation filter), Cyan (460-490 nmLED & 485/20 nm excitation filter), Green (535-600 nm LED & 549/15 nmexcitation filter), Green (535-600 nm LED & 560/25 nm excitationfilter), Red (620-750 nm LED & 650/13 nm excitation filter), and Near-IR(700-IR nm LED & 740/13 nm excitation filter). The two Green/excitationfilter combinations listed above can be provided optionally via, forexample, a mechanical flipper, when desiring to improve the brightnessof red and scarlet dyes. Of course, other LED bandwidths can also beused.

Referring now to FIG. 15, illustrated is an exemplary embodiment of asample vessel 222 having a plurality of light-emitting droplets 228where a real image 230 of the light-emitting droplets is identified bythe associated image sensor 114 of the displayed detector system.Disposed between the image sensor 114 and the sample vessel 222 is alens system 200. The lens system 200 makes the object space 204accessible for identifying light emitters by creating an image space206. In doing so, the operable distance of the image sensor 114 (i.e.,the object space 204) is increased in a direction perpendicular to thesurface of the image sensor 114. As shown in FIG. 15 (and as discussedin greater detail with respect to FIGS. 11 and 12), the lens system 200positioned between the image sensor 114 and sample vessel 222 creates areal image 230 of the light-emitting droplets 228 such that the imagesensor 114 can identify and count (and in some cases determine thethree-dimensional location of) each of the light-emitting droplets 228,based on their corresponding real image projection 230.

For example, the detector system (i.e., the lens system 200 and imagesensor 114) of FIG. 15 can be associated with a sample vessel 222 toidentify and count each light-emitting droplet 228 during or afterthermal cycling and/or to monitor the fluorescence intensity of eachlight-emitting droplet 228 during thermal cycling. As mentioned above,this may be done by simple identification of local peaks in the lightintensity values measured by the pixels. For ease of reference, thereal-image projections 230 of each light-emitting droplet 228 of FIG. 15have been reproduced in isolation within FIG. 16. As shown in FIG. 16,the real image 230 of the light-emitting droplet 228 is illustratedalong with corresponding cones of light recorded by a row of pixels ofthe associated image sensor and the resultant exemplary combined lightintensity profile 232.

The combined light intensity profile 232 is a combination of phenomenaillustrated and discussed above with respect to FIGS. 5-8. For example,similar to FIGS. 7 and 8, the local maxima light intensities 234 can befirst identified within the combined the light intensity profile 232 ofFIG. 16. These local maxima 234 each correspond to individuallight-emitting droplets and can be counted as such. Each of the lightprofiles associated with these light-emitting droplets can be subtractedfrom the combined light intensity profile 232 to allow for furtheridentification of local maxima in the revised light intensity profile(not shown) and thereby continue counting the total number oflight-emitting droplets. As described above, subtracting the lightprofiles of identified droplets from measured light intensity values canmake a subsequent identification of further fluorescent droplets moresensitive.

After all of the light-emitting droplets have been identified, thenumber of target molecules originally present in the sample vessel 222can be calculated. For embodiments in which there was statistically lessthan one target molecule in a droplet originally, the number of originaltarget molecules may be calculated in an approximation to be equal tothe number of light-emitting droplets. This approximation neglects thatoriginally, there may have been two or three or more target molecules ina light-emitting droplet. This approximation is better when the numberof target molecules is much smaller than the total number of droplets.This approximation can be improved using the Poisson distribution whichgives the probability distribution for one, two, or three or more targetmolecules in a light-emitting droplet depending on the averageprobability of having a target molecule in a droplet. For embodiments inwhich there was statistically more than one target molecule in a dropletoriginally, the number of target molecules in the droplets may beobtained by measuring the fluorescence intensity of each droplet aftereach thermal cycle and, for example, determine from the number of cyclesrequired to reach a certain intensity value the original amount oftarget molecules.

In an example, the total number of droplets may be computed by dividingthe total volume of water that has been used to create all droplets inthe sample emulsion by the average volume of the droplets. The totalnumber of droplets may also be measured using the control fluorophorediscussed above that provides a certain wavelength of fluorescence toall droplets. It should be appreciated that a control fluorophore may beprovided in a diluted concentration such that all droplets can beregistered and counted but droplets with amplified products of a certainconcentration have a greater fluorescence than droplets withoutamplified products. This can allow differentiation between the two typesof droplets. From the total number of droplets, the total volume oforiginal sample can be calculated by multiplying the total number ofdroplets with the average volume of droplets. From the number oforiginal target molecules and from the total volume of original sample,the original concentration of the target molecules can be obtained bydividing the number of original target molecules by the total volume oforiginal sample.

Alternatively, the certain wavelength may be different from thewavelength of the fluorophore indicating the presence of the amplifiedproduct so that droplets with amplified products can be identified bytheir color. This may require using an image sensor with a color filterarray with filters that have a transmission characteristic thatcoincides with the emission light of two different fluorophores, asdescribed above. Using a color filter array with two different filterscan beneficially allow for the identification of all droplets inaddition to those droplets that contain target amplicons. Furthermore,using different color filter arrays, it is possible to monitor multiplexPCR which uses different colors for different amplification products.

It should be appreciated that the resolution of the image sensor shouldbe high enough to distinguish between different droplets. For example,for droplets having a diameter of 2-5 μm, a pixel pitch of around 1 μmmay be used. Using a small sensor of, for example, 3 mm long by 3 mmwide, 9 million pixels are available for counting droplets. Thisexemplary image sensor would provide a pixel per droplet ratio of nearly200 for a sample divided into about 50,000 droplets.

In one embodiment, the density of droplets may be lower than the densityof the oil so that the droplets collect at the surface of the emulsion.In such an embodiment, the lens system may be a telecentric lens systemin which all magnification values are equal to 1. The telecentric lenssystem can be used to create images of the droplets at a distance fromthe image sensor at which droplets can be counted with the highestaccuracy. Assuming that the ratio of pixel per droplet is 200 (as above)and placing the images of the droplets at a distance from the imagesensor at which each image is seen by around 200 pixels on average(e.g., in an area of 14×14 pixels), it then follows that on average,most pixels record light from at most one droplet and only one droplet.It is clear that for such an example, the individual droplets should bevery easily be identifiable as local maxima in the measured lightintensity distribution. Accordingly, in some embodiments, the number ofdroplets, the droplet density, and the resolution of the image sensorcan be adjusted in combination with a lens system (e.g. a telecentriclens system) to more quickly and easily identify and countlight-emitting droplets. This can be particularly advantageous formultiplex PCR applications where a single droplet may be emittingmultiple wavelengths of light.

Embodiments of the present disclosure advantageously enable themonitoring of fluorescence intensity while the droplets are beingthermally cycled. This can provide a rich data source for calculatingthe relative or absolute amount of a target nucleic acid within asample. As shown in FIG. 17, is an exemplary data set illustrating themeasured fluorescence intensity of four different light emitters duringa given thermal cycling protocol. As shown, each of the light emittersincrease in fluorescence intensity at different rates, owing to thedifferent number of target nucleic acid templates originally captured bythe droplet.

In some embodiments, it can be advantageous to increase the probabilitythat each droplet will have more than one copy of the target nucleicacid. As shown in FIG. 17, light emitter 1 started with a single copy ofthe target nucleic acid, light emitter 2 started with two copies of thetarget nucleic acid, light emitter 3 started with 3 copies of the targetnucleic acid, and light emitter 4 started with 4 copies of the targetnucleic acid. The relative fluorescence for each light-emitting dropletover time is represented in FIG. 17 as a function of the exponentialgrowth of the template-derived amplicons. The data resolution providedby monitoring fluorescence intensity during thermal cycling is clearlysuperior to end-point fluorescence measurements, particularly in caseswhere the discernable difference between high fluorescence intensitiesbecomes nominal even though there was an obvious quantitative differencein earlier cycles.

Similar to those systems described in FIGS. 11 and 12, it should beappreciated that although specific reference was made to image sensor114 of FIGS. 2-4 when discussing the concepts illustrated within FIGS.13-15, other image sensors are compatible with—and can be used with—thelens systems described herein to form a detector system for identifyinga plurality of light emitting droplets and achieve the noted andimplicit benefits associated therewith. For example, the lens systemsdescribed herein can be associated with the image sensor 186 of FIGS. 9and 10 (or with any other image sensor described herein) and can form adetector system with the lens system 200 of FIG. 11 or the lens system210 of FIG. 12.

It is to be understood that features described with regard to thevarious embodiments herein may be mixed and matched in any desiredcombination. In addition, the concepts disclosed or envisioned hereinmay be embodied in other specific forms. The described embodiments areto be considered in all respects only as illustrative and notrestrictive. The scope is, therefore, indicated by the appended claimsrather than by the foregoing description. All changes which come withinthe meaning and range of equivalency of the claims are to be embracedwithin their scope.

1. An apparatus for identifying a plurality of light emitting droplets,comprising: a detector system comprising: an image sensor comprising anarray of pixels; and a structure associated with a surface of the imagesensor, the structure extending a height away from the surface of theimage sensor and defining a field of view for pixels within the array ofpixels; and a lens system positioned in front of the image sensor anddefining an object space of the detector system and an image space forthe image sensor, wherein light from a light emitting droplet that islocated in the object space is in the field of view of a plurality ofpixels within the array of pixels.
 2. The apparatus of claim 1, whereinthe lens system is configured to project a real image of the pluralityof light emitting droplets in the object space, the real image beinglocated within the image space of the lens system, and the image spacebeing positioned in front of the image sensor.
 3. The apparatus of claim1, wherein the detector system is configured to monitor a presence offluorescence within the plurality of light emitting droplets and tomonitor an intensity of fluorescence emitted by the plurality of lightemitting droplets, wherein the detector system monitors the fluorescenceemitted by the plurality of light emitting droplets by taking aplurality of images in a given time period so that a movement of eachone of the light emitting droplets is tracked and each one of the lightemitting droplets is identifiable over the given time period.
 4. Theapparatus of claim 1, further comprising a thermal cycler for cyclingthe plurality of light emitting droplets while the plurality of lightemitting droplets is located in the object space.
 5. The apparatus ofclaim 1, further comprising a light source configured to illuminate theobject space.
 6. The apparatus of claim 1, wherein the apparatuscomprises a plurality of detector systems and a plurality of objectspaces, wherein each object space is monitored by at least one detectorsystem.
 7. The apparatus of claim 6, wherein the plurality of objectspaces has a pitch of 9 mm or a multiple of 9 mm and a total number ofobject spaces is 96 or a divisor of 96 or wherein the plurality ofobject spaces has a pitch of 4.5 mm and the total number of objectspaces is
 384. 8. The apparatus of claim 1, wherein the lens systemcomprises a convergent lens system.
 9. The apparatus of claim 1, whereinthe lens system comprises a telecentric lens system.
 10. The apparatusof claim 9, wherein the telecentric lens system comprises two or moreconvergent lens systems, each adjacent pair of convergent lens systemsbeing spaced apart a distance equal to a sum of the focal lengths ofeach lens in each convergent lens system of the adjacent pair ofconvergent lens systems.
 11. The apparatus of claim 10, wherein eachadjacent pair of convergent lens systems comprises: a proximateconvergent lens system; and a distal convergent lens system, wherein theproximate convergent lens system is positioned between the image sensorand the distal convergent lens system, and wherein the focal length ofthe proximate convergent lens system is less than or equal to the focallength of the distal convergent lens system, wherein an axialmagnification of the adjacent pair of convergent lens systems increasesthe object space in a direction perpendicular to the image sensor. 12.The apparatus of claim 1, wherein the object space is extended between0.5 mm and 30 mm in a direction perpendicular to the image sensor. 13.The apparatus of claim 1, wherein the image space is limited in adirection perpendicular to the surface of the image sensor to a rangebetween 2 times the height of the structure and 1000 times the height ofthe structure and/or wherein the height of the structure is between 0.5and 3 times the pixel pitch.
 14. The apparatus of claim 1, wherein thestructure is associated with a surface of the image sensor and comprisesa plurality of walls positioned on pixel boundaries, forming a regulargrid, wherein the plurality of walls is formed around each pixel.
 15. Asystem for monitoring droplet polymerase chain reaction (PCR),comprising: an apparatus as in claim 1; an image processor configured toidentify a number of light-emitting droplets within the plurality ofdroplets; and a thermal cycler operably connected with the object space,wherein the object space defined by the lens system of the apparatuscomprises a plurality of droplets, a first subset of droplets of theplurality of droplets comprising PCR reagents, and a second subset ofdroplets of the plurality of droplets comprising a nucleic acid, andwherein the detector system of the apparatus is positioned such that areal image of the plurality of droplets is imagable by the image sensorin the image space.
 16. The system of claim 15, wherein the imageprocessor is additionally configured to count the number oflight-emitting droplets, wherein counting the number of light-emittingdroplets comprises capturing a light intensity value at eachphotoactivated pixel of the image sensor and identifying local maxima oflight intensity based on the light intensity values.
 17. The system ofclaim 16, wherein counting the number of light-emitting dropletscomprises: capturing a light intensity value at each photoactivatedpixel of the image sensor; identifying a first light-emitting droplet,comprising identifying a local maximum of light intensity and acorresponding light profile associated with the local maximum;subtracting the local maximum and the corresponding light profile fromthe captured light intensity values; and identifying subsequentlight-emitting droplets for each remaining local maximum andcorresponding light profile, wherein identifying the light profile forthe light emitting droplet comprises: calculating virtual lightintensity values for the first photoactivated pixel and the one or morephotoactivated pixels, wherein a virtual light intensity value for apixel represents the amount of light received by the pixel from avirtual light emitting droplet, the virtual light emitting droplethaving the same first photoactivated pixel as the light emittingdroplet; comparing the virtual light intensity values with the lightintensity values determined for light emitting droplet; and identifyingthe light profile for the light emitting droplet with the light profilefor the virtual light emitting droplet in the case that the virtuallight intensity values are fitting to the light intensity values for thelight emitting droplet within a tolerance level.
 18. The system of claim17, wherein the virtual light intensity value for the pixel iscalculated based on a light receiving area of the pixel receiving lightfrom the virtual light emitting droplet, wherein the virtual lightintensity value for the pixel is calculated further based on a solidangle of the light receiving area of the pixel with respect to thevirtual light emitting droplet.
 19. A method for identifyinglight-emitting droplets, comprising: thermal cycling a sample vesselcomprising a plurality of droplets, wherein the plurality of dropletscomprises droplet PCR reagents and a nucleic-acid-binding fluorophore,and wherein at least a subset of the plurality of droplets comprisenucleic acid; and monitoring the sample vessel using a detector system,the detector system comprising an image sensor with an array of pixels,a structure associated with a surface of the image sensor, the structureextending a height away from the surface of the image sensor anddefining a field of view for pixels within the array of pixels, and alens system positioned within the field of view and defining an objectspace of the detector system and an image space for the image sensor,wherein the sample vessel positions a sample within the object space ofthe detector system so that light from a light emitting droplet in thesample is in the field of view of a plurality of pixels within the arrayof pixels; capturing a light intensity value at each photoactivatedpixel of the image sensor receiving light from a real image of thesample vessel disposed in the image space; and identifying a number oflight-emitting droplets based on the light intensity values.
 20. Themethod as in claim 19, wherein identifying a number of light-emittingdroplets comprises: (i) determining a local maximum of light intensityand a corresponding light profile associated with the local maximum;(ii) subtracting the corresponding light profile associated with thelocal maximum from the captured light intensity values; and (iii)repeating steps (i) and (ii) for each subsequent local maximum andcorresponding light profile.